2022-01-21T19:20:57.602 INFO:root:teuthology version: 1.1.0-75f1bcd2 2022-01-21T19:20:57.752 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging 2022-01-21T19:20:57.754 INFO:teuthology.lock.ops:Updating [ubuntu@smithi084.front.sepia.ceph.com]: reset os type and version on server 2022-01-21T19:20:57.754 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2022-01-21T19:20:57.776 INFO:teuthology.lock.ops:Node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging is complete 2022-01-21T19:20:57.777 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi149.front.sepia.ceph.com' reimaging 2022-01-21T19:20:57.777 INFO:teuthology.lock.ops:Updating [ubuntu@smithi149.front.sepia.ceph.com]: reset os type and version on server 2022-01-21T19:20:57.778 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2022-01-21T19:20:57.798 INFO:teuthology.lock.ops:Node 'ubuntu@smithi149.front.sepia.ceph.com' reimaging is complete 2022-01-21T19:20:57.991 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of rhel 8.4 2022-01-21T19:20:58.013 INFO:teuthology.provision.fog.smithi149:Scheduling deploy of rhel 8.4 2022-01-21T19:20:58.305 INFO:teuthology.orchestra.console:Power off smithi084 2022-01-21T19:20:58.319 INFO:teuthology.orchestra.console:Power off smithi149 2022-01-21T19:21:10.797 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2022-01-21T19:21:10.804 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2022-01-21T19:21:10.899 INFO:teuthology.orchestra.console:Power on smithi084 2022-01-21T19:21:10.907 INFO:teuthology.orchestra.console:Power on smithi149 2022-01-21T19:21:15.153 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2022-01-21T19:21:15.164 INFO:teuthology.orchestra.console:Power on for smithi149 completed 2022-01-21T19:21:15.255 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2022-01-21T19:21:15.265 INFO:teuthology.provision.fog.smithi149:Waiting for deploy to finish 2022-01-21T19:25:47.011 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-21T19:25:47.222 INFO:teuthology.provision.fog.smithi084:Node is ready 2022-01-21T19:25:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2022-01-21T19:25:47.293 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2022-01-21T19:25:47.628 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2022-01-21T19:25:56.084 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-21T19:25:56.303 INFO:teuthology.provision.fog.smithi149:Node is ready 2022-01-21T19:25:56.319 INFO:teuthology.orchestra.run.smithi149.stdout:smithi149.front.sepia.ceph.com 2022-01-21T19:25:56.374 INFO:teuthology.orchestra.run.smithi149.stdout:172.21.15.149 smithi149.front.sepia.ceph.com smithi149 2022-01-21T19:25:56.720 INFO:teuthology.provision.fog.smithi149:Deploy complete! 2022-01-21T19:25:56.866 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2022-01-21T19:25:56.867 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-21T19:25:56.887 INFO:teuthology.lock.ops:Checking smithi149.front.sepia.ceph.com 2022-01-21T19:25:56.888 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-21T19:25:56.906 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2022-01-21T19:25:56.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-01-21T19:25:57.138 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2022-01-21T19:25:57.193 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Red Hat Enterprise Linux" 2022-01-21T19:25:57.194 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8.4 (Ootpa)" 2022-01-21T19:25:57.194 INFO:teuthology.orchestra.run.smithi084.stdout:ID="rhel" 2022-01-21T19:25:57.194 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="fedora" 2022-01-21T19:25:57.195 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8.4" 2022-01-21T19:25:57.195 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2022-01-21T19:25:57.195 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-01-21T19:25:57.195 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2022-01-21T19:25:57.196 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-01-21T19:25:57.196 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.redhat.com/" 2022-01-21T19:25:57.196 INFO:teuthology.orchestra.run.smithi084.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-01-21T19:25:57.196 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-21T19:25:57.197 INFO:teuthology.orchestra.run.smithi084.stdout: 2022-01-21T19:25:57.197 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-21T19:25:57.197 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-01-21T19:25:57.197 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-01-21T19:25:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-01-21T19:25:57.199 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2022-01-21T19:25:57.218 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: set os type and version on server 2022-01-21T19:25:57.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2022-01-21T19:25:57.437 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2022-01-21T19:25:57.491 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="Red Hat Enterprise Linux" 2022-01-21T19:25:57.492 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8.4 (Ootpa)" 2022-01-21T19:25:57.492 INFO:teuthology.orchestra.run.smithi149.stdout:ID="rhel" 2022-01-21T19:25:57.492 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="fedora" 2022-01-21T19:25:57.493 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8.4" 2022-01-21T19:25:57.493 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2022-01-21T19:25:57.493 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-01-21T19:25:57.494 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2022-01-21T19:25:57.494 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-01-21T19:25:57.494 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://www.redhat.com/" 2022-01-21T19:25:57.494 INFO:teuthology.orchestra.run.smithi149.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-01-21T19:25:57.494 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-21T19:25:57.495 INFO:teuthology.orchestra.run.smithi149.stdout: 2022-01-21T19:25:57.495 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-21T19:25:57.495 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-01-21T19:25:57.495 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-01-21T19:25:57.496 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-01-21T19:25:57.496 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2022-01-21T19:25:57.602 INFO:teuthology.dispatcher.supervisor:Running job 6632348 2022-01-21T19:25:57.602 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_75f1bcd237a8b6022305cd53ba87e106d45aa665/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-01-21_15:31:49-rgw-wip-yuri7-testing-2022-01-20-1609-distro-default-smithi/6632348 --name yuriw-2022-01-21_15:31:49-rgw-wip-yuri7-testing-2022-01-20-1609-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} -- /home/teuthworker/archive/yuriw-2022-01-21_15:31:49-rgw-wip-yuri7-testing-2022-01-20-1609-distro-default-smithi/6632348/orig.config.yaml 2022-01-21T19:25:57.606 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-01-21_15:31:49-rgw-wip-yuri7-testing-2022-01-20-1609-distro-default-smithi/6632348 2022-01-21T19:25:57.607 INFO:teuthology.dispatcher.supervisor:Job PID: 29064 2022-01-21T19:25:57.607 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-01-22T01:56:26.171 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-01-22T01:56:26.378 INFO:teuthology.kill:Killing Pids: {29064} 2022-01-22T01:56:27.510 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-01-22T01:56:27.512 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-01-22T01:56:27.512 INFO:teuthology.misc:Compressing logs... 2022-01-22T01:56:27.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-01-22T01:56:27.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2022-01-22T01:56:28.122 INFO:teuthology.misc:Compressing logs... 2022-01-22T01:56:28.276 INFO:teuthology.misc:Compressing logs... 2022-01-22T02:00:46.102 INFO:teuthology.kill:No teuthology processes running 2022-01-22T02:00:47.234 INFO:teuthology.kill:Nuking machines: ['smithi084', 'smithi149'] 2022-01-22T02:00:48.220 INFO:teuthology.kill:2022-01-22 02:00:48,220.220 INFO:teuthology.nuke:targets: 2022-01-22T02:00:48.221 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCynlfk1Xgt43WzOmjidfFYeQUkboNXRBZ/+OhgaH94nFMXMjk80RboQUjHgD84YrVVxOeQHF81lqYD+Q86AW+uOKxSqNVtkabEJNp2J9J5ej7gCx1FUv5ASPFxci9jChMfP/BYH/Apa3oOunsvu7zTXLZZs7LmmTipnSqdN69eiu5gJ2z3jXNIj5j/rzVLJC6tG1AOYMMQ5gNQv2y6rW77Jkolg1Y943JS1loGfQ7BJQTNFJWErX4cDMj7/Z7xXRC2y6z7Y73/RNTwNXLbIEUpqsXU1HR7LSbjDy56gxojpveHnFR6cC0u1krS5uOvS0kLhxDjAb6m6uSunPaYd4xqCdfy3FplmxO3M8C2dXckAeQGPbLgD3kzXnFwdFyOHUBpnTfn+n0AOgNdDsPLO+adAeGN3JHNyy6RUl3296s4bNpax+3/6oKFhrA/6IJkeIMjxdDVX1qL4L/pdAPfTRdOL0zDsrBIYJ4Sk4dhV8a4+bHweIWl/BFNBdqSSFqyz70= 2022-01-22T02:00:48.222 INFO:teuthology.kill: smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlfcZlCKhtip9qWnJnEGb5RYNazoarg7Pwl1VaPKZn3/GI+MfRmAbMxO35ECMvFV96041BgjFu6MGGoeTmmNYYxkd5CGaZryyMFlOvnjF8WOz3ydn8sLWJHD0g3uWxqQ0WyR/buMFyalhciWf4dUrJjM/RQzdt6vt4uSxve7O8M2As+7Z7h+OtD3NyjpJCA8W2Y1onAa47m/nBY47QHAsT3ri7P6VewXgRmUGMVlgGJ6oaMNMnCSNx6p++RMy/gXxjROy8QyZ5HOl2x/rYvp2/o21fcDfWIHAtgwj+E4D92tMPm5MtXPNRws+4srdXHZ/2Fw43DNguIzYu/gggpany6+cdncG/ixlzQ7vfgmdnCPaVYx4eulrLdHZk3Gmlmb0LAcd3ZxHn1P7S9NdNHgGYkvmC2SvTJbl/S2IrsAoOliERIvc4TUrMxQ1u+OO9EEguC5aBzvbw4HaCDYOcYPHmvc/0Z8nvc57XEr92UnBnHKVrS8AxCsa3CXB3FEZbYWE= 2022-01-22T02:00:48.247 INFO:teuthology.kill:2022-01-22 02:00:48,246.246 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-22T02:00:48.250 INFO:teuthology.kill:2022-01-22 02:00:48,250.250 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-22T02:00:48.326 INFO:teuthology.kill:2022-01-22 02:00:48,325.325 INFO:teuthology.orchestra.console:Power off smithi084 2022-01-22T02:00:48.339 INFO:teuthology.kill:2022-01-22 02:00:48,338.338 INFO:teuthology.orchestra.console:Power off smithi149 2022-01-22T02:01:00.805 INFO:teuthology.kill:2022-01-22 02:01:00,804.804 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2022-01-22T02:01:00.810 INFO:teuthology.kill:2022-01-22 02:01:00,810.810 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2022-01-22T02:01:00.953 INFO:teuthology.kill:2022-01-22 02:01:00,953.953 INFO:teuthology.lock.ops:unlocked: smithi084.front.sepia.ceph.com 2022-01-22T02:01:00.956 INFO:teuthology.kill:2022-01-22 02:01:00,956.956 INFO:teuthology.lock.ops:unlocked: smithi149.front.sepia.ceph.com 2022-01-22T02:03:01.260 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-01-22T02:03:01.359 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi149 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-01-21_15:32:46-rbd-wip-yuri7-testing-2022-01-20-1609-distro-default-smithi/6632506