2023-02-11T09:41:10.033 INFO:root:teuthology version: 0.0.1.dev37+g00690ec
2023-02-11T09:41:10.231 INFO:teuthology.lock.ops:Start node 'smithi007.front.sepia.ceph.com' reimaging
2023-02-11T09:41:10.232 INFO:teuthology.lock.ops:Updating [smithi007.front.sepia.ceph.com]: reset os type and version on server
2023-02-11T09:41:10.232 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server
2023-02-11T09:41:10.267 INFO:teuthology.lock.ops:Node 'smithi007.front.sepia.ceph.com' reimaging is complete
2023-02-11T09:41:10.267 INFO:teuthology.lock.ops:Start node 'smithi132.front.sepia.ceph.com' reimaging
2023-02-11T09:41:10.267 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: reset os type and version on server
2023-02-11T09:41:10.267 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server
2023-02-11T09:41:10.404 INFO:teuthology.lock.ops:Node 'smithi132.front.sepia.ceph.com' reimaging is complete
2023-02-11T09:41:10.437 INFO:teuthology.provision.fog.smithi007:Scheduling deploy of rhel 8.4
2023-02-11T09:41:10.558 INFO:teuthology.provision.fog.smithi132:Scheduling deploy of rhel 8.4
2023-02-11T09:41:10.648 INFO:teuthology.orchestra.console:Power off smithi007
2023-02-11T09:41:10.752 INFO:teuthology.orchestra.console:Power off smithi132
2023-02-11T09:41:19.229 INFO:teuthology.orchestra.console:Power off for smithi007 completed
2023-02-11T09:41:19.330 INFO:teuthology.orchestra.console:Power on smithi007
2023-02-11T09:41:23.337 INFO:teuthology.orchestra.console:Power off for smithi132 completed
2023-02-11T09:41:23.438 INFO:teuthology.orchestra.console:Power on smithi132
2023-02-11T09:41:27.697 INFO:teuthology.orchestra.console:Power on for smithi132 completed
2023-02-11T09:41:27.800 INFO:teuthology.provision.fog.smithi132:Waiting for deploy to finish
2023-02-11T09:41:27.804 INFO:teuthology.orchestra.console:Power on for smithi007 completed
2023-02-11T09:41:27.905 INFO:teuthology.provision.fog.smithi007:Waiting for deploy to finish
2023-02-11T09:43:58.682 ERROR:teuthology.orchestra.connection:Error authenticating with smithi132.front.sepia.ceph.com: Authentication failed.
2023-02-11T09:44:17.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.7
2023-02-11T09:44:18.181 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-02-11T09:44:26.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.7
2023-02-11T09:44:27.237 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-02-11T09:44:35.269 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.7
2023-02-11T09:44:36.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-02-11T09:44:44.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.7
2023-02-11T09:44:45.349 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-02-11T09:45:50.352 WARNING:teuthology.provision.fog:timed out
2023-02-11T09:45:51.352 WARNING:teuthology.provision.fog:timed out
2023-02-11T09:45:56.851 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-02-11T09:45:57.133 INFO:teuthology.provision.fog.smithi007:Node is ready
2023-02-11T09:45:57.147 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007.front.sepia.ceph.com
2023-02-11T09:45:57.202 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7 smithi007.front.sepia.ceph.com smithi007
2023-02-11T09:45:57.559 INFO:teuthology.provision.fog.smithi007:Deploy complete!
2023-02-11T09:45:57.698 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-02-11T09:45:57.965 INFO:teuthology.provision.fog.smithi132:Node is ready
2023-02-11T09:45:58.019 INFO:teuthology.orchestra.run.smithi132.stdout:smithi132.front.sepia.ceph.com
2023-02-11T09:45:58.074 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132 smithi132.front.sepia.ceph.com smithi132
2023-02-11T09:45:58.387 INFO:teuthology.provision.fog.smithi132:Deploy complete!
2023-02-11T09:45:58.686 INFO:teuthology.lock.ops:Checking smithi007.front.sepia.ceph.com
2023-02-11T09:45:58.686 INFO:teuthology.lock.ops:New key found. Updating...
2023-02-11T09:45:58.730 INFO:teuthology.lock.ops:Checking smithi132.front.sepia.ceph.com
2023-02-11T09:45:58.730 INFO:teuthology.lock.ops:New key found. Updating...
2023-02-11T09:45:58.776 INFO:teuthology.lock.ops:Updating [smithi007.front.sepia.ceph.com]: set os type and version on server
2023-02-11T09:45:58.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com'
2023-02-11T09:45:58.962 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64
2023-02-11T09:45:59.017 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Red Hat Enterprise Linux"
2023-02-11T09:45:59.017 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="8.4 (Ootpa)"
2023-02-11T09:45:59.017 INFO:teuthology.orchestra.run.smithi007.stdout:ID="rhel"
2023-02-11T09:45:59.017 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="fedora"
2023-02-11T09:45:59.017 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="8.4"
2023-02-11T09:45:59.017 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el8"
2023-02-11T09:45:59.017 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-02-11T09:45:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31"
2023-02-11T09:45:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-02-11T09:45:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.redhat.com/"
2023-02-11T09:45:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-02-11T09:45:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-02-11T09:45:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:
2023-02-11T09:45:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-02-11T09:45:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-02-11T09:45:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-02-11T09:45:59.019 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-02-11T09:45:59.020 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server
2023-02-11T09:45:59.048 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: set os type and version on server
2023-02-11T09:45:59.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com'
2023-02-11T09:45:59.242 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64
2023-02-11T09:45:59.300 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="Red Hat Enterprise Linux"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="8.4 (Ootpa)"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:ID="rhel"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="fedora"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="8.4"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el8"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://www.redhat.com/"
2023-02-11T09:45:59.301 INFO:teuthology.orchestra.run.smithi132.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-02-11T09:45:59.302 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-02-11T09:45:59.302 INFO:teuthology.orchestra.run.smithi132.stdout:
2023-02-11T09:45:59.302 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-02-11T09:45:59.302 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-02-11T09:45:59.302 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-02-11T09:45:59.302 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-02-11T09:45:59.303 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server
2023-02-11T09:45:59.431 INFO:teuthology.dispatcher.supervisor:Running job 7160538
2023-02-11T09:45:59.432 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7160538 --name yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_mon_osdmap_prune} -- /home/teuthworker/archive/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7160538/orig.config.yaml
2023-02-11T09:45:59.437 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7160538
2023-02-11T09:45:59.438 INFO:teuthology.dispatcher.supervisor:Job PID: 31890
2023-02-11T09:45:59.438 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-02-11T21:46:27.301 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-02-11T21:46:31.075 INFO:teuthology.kill:Killing Pids: {31890}
2023-02-11T21:46:46.309 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2']
2023-02-11T21:46:46.310 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']
2023-02-11T21:46:46.318 INFO:teuthology.misc:Compressing logs...
2023-02-11T21:46:46.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com'
2023-02-11T21:46:48.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com'
2023-02-11T21:46:49.716 INFO:teuthology.kill:No teuthology processes running
2023-02-11T21:46:50.965 INFO:teuthology.kill:Nuking machines: ['smithi007', 'smithi132']
2023-02-11T21:46:52.165 INFO:teuthology.kill:2023-02-11 21:46:52,164.164 INFO:teuthology.nuke:targets:
2023-02-11T21:46:52.165 INFO:teuthology.kill:  smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCl8efc8HmAkABT4VRzzB7Slrosw8hyTVp7prMgtisctkMToLdzxMX2jE2mFfhPmiSc3v2+ydCwCccznTXjXkitbJX9gS6FfaUGu2IQXRCuufEgVmJgHjkUPYLytrIYV6l9LHJb0X7ZDcZLuQLgag5Wi9z+fAFwGl3bwoP23U8Ym80nXtXhb56ZacHLI/Co8cfYBfDo1hhBu3qN4VZYBxQQ5tZhOa31LuvxlIw56nykjHyONsEeFSFe/rNt7fCxrycNW3u1UL5j0Pj/PRvjTT24yDdMMYc6TdZg0rUs5YIKIShbjsGWSjhX02S3R+AZ+H9B8tQdmgSsd4A0uCCaDriWX53VfZz7Fe++i346frILUeORJ3gqllkhzdYZfKi5g0X9xT7+0bNIxY3kd/r2JtuQV8dFcsZQ5OfEfeZoVad1hIa+ZhlDCd/4zM35qOfC+YL/piNHq7RPp/GNB0eQbyj8EEbeZfHoJG+gP3TJGvgnG2ZxoEcW8X169jE0nS/l9Xs=
2023-02-11T21:46:52.165 INFO:teuthology.kill:  smithi132.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC76z+CwKCKcgAscM7qaXcWtpTr5bQLDIl/0JvNrPCmzX0U8Lk6NNRAGxZOCuCgtRqcep0PiU9YDfb827MD+l/PSPtsfbe3lBCunpaoayVi19yLP3HoZu27x7SMHXiSG55gSrq0GHHVCMdOQvWUm0/Js9jt/WZQLENmd8YCzGzKMtINq80U0woSztcd4eHNyZFAEu61pVYiMiPiyRf9v6W2w+nfujeWBnC9jSS3If7Fm4kFDBR5a2HSB5QFdkfDspSP1Y5VE86aiufPG4HH+/Yr2qz9lQzKvNIHLyZQ0Z3SpX6+Y6nnwKQSE+ZMwy6/vfaWqtegjzAklzcLI+SQ0Up74grW2MZAcQQYC5lYR/M74f5Q5/FzjeyA6V9MwLkuYT4rtw3x/9fG0XWXYLijO8K+0KoVvGqbPH/qB9qhmTN3ABEkhDItUQLWJEgbaRWU1dtrtsfv4B/Jeo5xH5xZ3pThKme+isnDjkql53Wdq2OJWvFJqWctG9Nu6AppCts5qVE=
2023-02-11T21:46:52.165 INFO:teuthology.kill:2023-02-11 21:46:52,164.164 INFO:teuthology.nuke:Checking targets against current locks
2023-02-11T21:46:52.213 INFO:teuthology.kill:2023-02-11 21:46:52,213.213 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-02-11T21:46:52.233 INFO:teuthology.kill:2023-02-11 21:46:52,232.232 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-02-11T21:46:52.297 INFO:teuthology.kill:2023-02-11 21:46:52,297.297 INFO:teuthology.orchestra.console:Power off smithi007
2023-02-11T21:46:52.371 INFO:teuthology.kill:2023-02-11 21:46:52,371.371 INFO:teuthology.orchestra.console:Power off smithi132
2023-02-11T21:47:00.876 INFO:teuthology.kill:2023-02-11 21:47:00,875.875 INFO:teuthology.orchestra.console:Power off for smithi007 completed
2023-02-11T21:47:00.884 INFO:teuthology.kill:2023-02-11 21:47:00,884.884 INFO:teuthology.orchestra.console:Power off for smithi132 completed
2023-02-11T21:47:01.020 INFO:teuthology.kill:2023-02-11 21:47:01,020.020 INFO:teuthology.lock.ops:unlocked: smithi007.front.sepia.ceph.com
2023-02-11T21:47:01.023 INFO:teuthology.kill:2023-02-11 21:47:01,023.023 INFO:teuthology.lock.ops:unlocked: smithi132.front.sepia.ceph.com
2023-02-11T21:49:01.280 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-02-11T21:49:01.524 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi132 but it was locked by another job: /home/teuthworker/archive/cbodley-2023-02-11_16:16:51-rgw-wip-cbodley-testing-distro-default-smithi/7169658