2023-02-16T14:48:12.446 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5
2023-02-16T14:48:12.566 INFO:teuthology.lock.ops:Start node 'smithi078.front.sepia.ceph.com' reimaging
2023-02-16T14:48:12.568 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: reset os type and version on server
2023-02-16T14:48:12.568 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server
2023-02-16T14:48:12.590 INFO:teuthology.lock.ops:Node 'smithi078.front.sepia.ceph.com' reimaging is complete
2023-02-16T14:48:12.590 INFO:teuthology.lock.ops:Start node 'smithi103.front.sepia.ceph.com' reimaging
2023-02-16T14:48:12.591 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: reset os type and version on server
2023-02-16T14:48:12.591 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server
2023-02-16T14:48:12.609 INFO:teuthology.lock.ops:Node 'smithi103.front.sepia.ceph.com' reimaging is complete
2023-02-16T14:48:12.736 INFO:teuthology.provision.fog.smithi078:Scheduling deploy of rhel 8.6
2023-02-16T14:48:12.765 INFO:teuthology.provision.fog.smithi103:Scheduling deploy of rhel 8.6
2023-02-16T14:48:12.944 INFO:teuthology.orchestra.console:Power off smithi078
2023-02-16T14:48:12.983 INFO:teuthology.orchestra.console:Power off smithi103
2023-02-16T14:48:21.347 INFO:teuthology.orchestra.console:Power off for smithi103 completed
2023-02-16T14:48:21.448 INFO:teuthology.orchestra.console:Power on smithi103
2023-02-16T14:48:25.431 INFO:teuthology.orchestra.console:Power off for smithi078 completed
2023-02-16T14:48:25.532 INFO:teuthology.orchestra.console:Power on smithi078
2023-02-16T14:48:29.789 INFO:teuthology.orchestra.console:Power on for smithi078 completed
2023-02-16T14:48:29.815 INFO:teuthology.orchestra.console:Power on for smithi103 completed
2023-02-16T14:48:29.891 INFO:teuthology.provision.fog.smithi078:Waiting for deploy to finish
2023-02-16T14:48:29.916 INFO:teuthology.provision.fog.smithi103:Waiting for deploy to finish
2023-02-16T14:51:18.917 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-02-16T14:51:27.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-02-16T14:51:33.989 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-02-16T14:51:37.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-02-16T14:51:43.045 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-02-16T14:51:46.089 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-02-16T14:51:52.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-02-16T14:51:55.177 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-02-16T14:52:01.161 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-02-16T14:52:04.233 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-02-16T14:52:10.249 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-02-16T14:52:13.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-02-16T14:52:19.337 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-02-16T14:52:28.425 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-02-16T14:53:19.365 WARNING:teuthology.provision.fog:timed out
2023-02-16T14:53:25.723 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-02-16T14:53:26.053 INFO:teuthology.provision.fog.smithi078:Node is ready
2023-02-16T14:53:26.072 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078
2023-02-16T14:53:26.132 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78 smithi078.front.sepia.ceph.com smithi078
2023-02-16T14:53:26.506 INFO:teuthology.provision.fog.smithi078:Deploy complete!
2023-02-16T14:53:34.441 WARNING:teuthology.provision.fog:timed out
2023-02-16T14:53:40.769 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-02-16T14:53:41.033 INFO:teuthology.provision.fog.smithi103:Node is ready
2023-02-16T14:53:41.049 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103
2023-02-16T14:53:41.106 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103 smithi103.front.sepia.ceph.com smithi103
2023-02-16T14:53:41.460 INFO:teuthology.provision.fog.smithi103:Deploy complete!
2023-02-16T14:53:41.633 INFO:teuthology.lock.ops:Checking smithi078.front.sepia.ceph.com
2023-02-16T14:53:41.634 INFO:teuthology.lock.ops:New key found. Updating...
2023-02-16T14:53:41.663 INFO:teuthology.lock.ops:Checking smithi103.front.sepia.ceph.com
2023-02-16T14:53:41.664 INFO:teuthology.lock.ops:New key found. Updating...
2023-02-16T14:53:41.719 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: set os type and version on server
2023-02-16T14:53:41.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com'
2023-02-16T14:53:41.976 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64
2023-02-16T14:53:42.034 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Red Hat Enterprise Linux"
2023-02-16T14:53:42.035 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8.6 (Ootpa)"
2023-02-16T14:53:42.035 INFO:teuthology.orchestra.run.smithi078.stdout:ID="rhel"
2023-02-16T14:53:42.035 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="fedora"
2023-02-16T14:53:42.035 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8.6"
2023-02-16T14:53:42.035 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8"
2023-02-16T14:53:42.036 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-02-16T14:53:42.036 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31"
2023-02-16T14:53:42.036 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-02-16T14:53:42.036 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.redhat.com/"
2023-02-16T14:53:42.036 INFO:teuthology.orchestra.run.smithi078.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-02-16T14:53:42.036 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-02-16T14:53:42.036 INFO:teuthology.orchestra.run.smithi078.stdout:
2023-02-16T14:53:42.036 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-02-16T14:53:42.037 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-02-16T14:53:42.037 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-02-16T14:53:42.037 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-02-16T14:53:42.038 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server
2023-02-16T14:53:42.080 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: set os type and version on server
2023-02-16T14:53:42.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-02-16T14:53:42.224 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64
2023-02-16T14:53:42.281 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="Red Hat Enterprise Linux"
2023-02-16T14:53:42.282 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="8.6 (Ootpa)"
2023-02-16T14:53:42.282 INFO:teuthology.orchestra.run.smithi103.stdout:ID="rhel"
2023-02-16T14:53:42.282 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="fedora"
2023-02-16T14:53:42.282 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="8.6"
2023-02-16T14:53:42.282 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el8"
2023-02-16T14:53:42.282 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-02-16T14:53:42.282 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31"
2023-02-16T14:53:42.283 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-02-16T14:53:42.283 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://www.redhat.com/"
2023-02-16T14:53:42.283 INFO:teuthology.orchestra.run.smithi103.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-02-16T14:53:42.283 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-02-16T14:53:42.283 INFO:teuthology.orchestra.run.smithi103.stdout:
2023-02-16T14:53:42.283 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-02-16T14:53:42.283 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-02-16T14:53:42.283 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-02-16T14:53:42.284 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-02-16T14:53:42.284 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server
2023-02-16T14:53:42.439 INFO:teuthology.dispatcher.supervisor:Running job 7176268
2023-02-16T14:53:42.440 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/bin/teuthology -v --owner scheduled_cbodley@teuthology --archive /home/teuthworker/archive/cbodley-2023-02-16_14:01:19-rgw-wip-cbodley-testing-distro-default-smithi/7176268 --name cbodley-2023-02-16_14:01:19-rgw-wip-cbodley-testing-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} -- /home/teuthworker/archive/cbodley-2023-02-16_14:01:19-rgw-wip-cbodley-testing-distro-default-smithi/7176268/orig.config.yaml
2023-02-16T14:53:42.443 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/cbodley-2023-02-16_14:01:19-rgw-wip-cbodley-testing-distro-default-smithi/7176268
2023-02-16T14:53:42.444 INFO:teuthology.dispatcher.supervisor:Job PID: 21164
2023-02-16T14:53:42.444 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-02-17T02:54:40.716 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-02-17T02:54:44.897 INFO:teuthology.kill:Killing Pids: {21164}
2023-02-17T02:54:46.374 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1']
2023-02-17T02:54:46.375 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']
2023-02-17T02:54:46.398 INFO:teuthology.misc:Compressing logs...
2023-02-17T02:54:46.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com'
2023-02-17T02:54:48.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-02-17T02:54:49.063 INFO:teuthology.misc:Compressing logs...
2023-02-17T02:54:49.305 INFO:teuthology.misc:Compressing logs...
2023-02-17T03:03:58.953 INFO:teuthology.kill:No teuthology processes running
2023-02-17T03:04:00.242 INFO:teuthology.kill:Nuking machines: ['smithi078', 'smithi103']
2023-02-17T03:04:00.874 INFO:teuthology.kill:2023-02-17 03:04:00,874.874 INFO:teuthology.nuke:targets:
2023-02-17T03:04:00.888 INFO:teuthology.kill:  smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-02-17T03:04:00.889 INFO:teuthology.kill:  smithi103.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-02-17T03:04:00.889 INFO:teuthology.kill:2023-02-17 03:04:00,874.874 INFO:teuthology.nuke:Checking targets against current locks
2023-02-17T03:04:00.932 INFO:teuthology.kill:2023-02-17 03:04:00,932.932 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-02-17T03:04:00.961 INFO:teuthology.kill:2023-02-17 03:04:00,961.961 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-02-17T03:04:01.018 INFO:teuthology.kill:2023-02-17 03:04:01,018.018 INFO:teuthology.orchestra.console:Power off smithi078
2023-02-17T03:04:01.059 INFO:teuthology.kill:2023-02-17 03:04:01,058.058 INFO:teuthology.orchestra.console:Power off smithi103
2023-02-17T03:04:13.512 INFO:teuthology.kill:2023-02-17 03:04:13,511.511 INFO:teuthology.orchestra.console:Power off for smithi078 completed
2023-02-17T03:04:13.536 INFO:teuthology.kill:2023-02-17 03:04:13,536.536 INFO:teuthology.orchestra.console:Power off for smithi103 completed
2023-02-17T03:04:13.734 INFO:teuthology.kill:2023-02-17 03:04:13,734.734 INFO:teuthology.lock.ops:unlocked: smithi078.front.sepia.ceph.com
2023-02-17T03:04:13.745 INFO:teuthology.kill:2023-02-17 03:04:13,744.744 INFO:teuthology.lock.ops:unlocked: smithi103.front.sepia.ceph.com
2023-02-17T03:06:14.106 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-02-17T03:06:14.315 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi078 but it was locked by another job: /home/teuthworker/archive/dis-2023-02-17_00:48:46-rbd-wip-dis-testing-distro-default-smithi/7177953
2023-02-17T03:06:14.316 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi103 but it was locked by another job: /home/teuthworker/archive/dis-2023-02-17_00:48:46-rbd-wip-dis-testing-distro-default-smithi/7177953