2023-02-11T16:06:43.786 INFO:root:teuthology version: 0.0.1.dev37+g00690ec 2023-02-11T16:06:43.954 INFO:teuthology.lock.ops:Start node 'smithi040.front.sepia.ceph.com' reimaging 2023-02-11T16:06:43.954 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: reset os type and version on server 2023-02-11T16:06:43.954 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-02-11T16:06:43.971 INFO:teuthology.lock.ops:Node 'smithi040.front.sepia.ceph.com' reimaging is complete 2023-02-11T16:06:43.971 INFO:teuthology.lock.ops:Start node 'smithi086.front.sepia.ceph.com' reimaging 2023-02-11T16:06:43.971 INFO:teuthology.lock.ops:Updating [smithi086.front.sepia.ceph.com]: reset os type and version on server 2023-02-11T16:06:43.971 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2023-02-11T16:06:43.990 INFO:teuthology.lock.ops:Node 'smithi086.front.sepia.ceph.com' reimaging is complete 2023-02-11T16:06:44.117 INFO:teuthology.provision.fog.smithi040:Scheduling deploy of centos 8.stream 2023-02-11T16:06:44.140 INFO:teuthology.provision.fog.smithi086:Scheduling deploy of centos 8.stream 2023-02-11T16:06:44.362 INFO:teuthology.orchestra.console:Power off smithi040 2023-02-11T16:06:44.380 INFO:teuthology.orchestra.console:Power off smithi086 2023-02-11T16:06:44.624 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2023-02-11T16:06:44.633 INFO:teuthology.orchestra.console:Power off for smithi040 completed 2023-02-11T16:06:44.725 INFO:teuthology.orchestra.console:Power on smithi086 2023-02-11T16:06:44.735 INFO:teuthology.orchestra.console:Power on smithi040 2023-02-11T16:06:48.992 INFO:teuthology.orchestra.console:Power on for smithi040 completed 2023-02-11T16:06:49.092 INFO:teuthology.provision.fog.smithi040:Waiting for deploy to finish 2023-02-11T16:06:53.252 INFO:teuthology.orchestra.console:Power on for smithi086 completed 2023-02-11T16:06:53.355 INFO:teuthology.provision.fog.smithi086:Waiting for deploy to finish 2023-02-11T16:09:53.158 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-02-11T16:09:57.509 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86 2023-02-11T16:10:02.245 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-02-11T16:10:06.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86 2023-02-11T16:10:11.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-02-11T16:10:15.621 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86 2023-02-11T16:10:24.709 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86 2023-02-11T16:10:33.797 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86 2023-02-11T16:11:17.338 WARNING:teuthology.provision.fog:timed out 2023-02-11T16:11:23.637 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-02-11T16:11:23.936 INFO:teuthology.provision.fog.smithi040:Node is ready 2023-02-11T16:11:23.996 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040.front.sepia.ceph.com 2023-02-11T16:11:24.056 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40 smithi040.front.sepia.ceph.com smithi040 2023-02-11T16:11:24.431 INFO:teuthology.provision.fog.smithi040:Deploy complete! 2023-02-11T16:11:39.849 WARNING:teuthology.provision.fog:timed out 2023-02-11T16:11:46.184 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-02-11T16:11:46.494 INFO:teuthology.provision.fog.smithi086:Node is ready 2023-02-11T16:11:46.511 INFO:teuthology.orchestra.run.smithi086.stdout:smithi086.front.sepia.ceph.com 2023-02-11T16:11:46.570 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.15.86 smithi086.front.sepia.ceph.com smithi086 2023-02-11T16:11:46.918 INFO:teuthology.provision.fog.smithi086:Deploy complete! 2023-02-11T16:11:47.202 INFO:teuthology.lock.ops:Checking smithi040.front.sepia.ceph.com 2023-02-11T16:11:47.209 INFO:teuthology.lock.ops:New key found. Updating... 2023-02-11T16:11:47.225 INFO:teuthology.lock.ops:Checking smithi086.front.sepia.ceph.com 2023-02-11T16:11:47.225 INFO:teuthology.lock.ops:New key found. Updating... 2023-02-11T16:11:47.240 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: set os type and version on server 2023-02-11T16:11:47.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-02-11T16:11:47.493 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2023-02-11T16:11:47.551 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2023-02-11T16:11:47.551 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8" 2023-02-11T16:11:47.551 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2023-02-11T16:11:47.551 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2023-02-11T16:11:47.551 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8" 2023-02-11T16:11:47.552 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2023-02-11T16:11:47.552 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 8" 2023-02-11T16:11:47.552 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2023-02-11T16:11:47.552 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-02-11T16:11:47.552 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2023-02-11T16:11:47.552 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-02-11T16:11:47.552 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-02-11T16:11:47.552 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-02-11T16:11:47.553 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-02-11T16:11:47.569 INFO:teuthology.lock.ops:Updating [smithi086.front.sepia.ceph.com]: set os type and version on server 2023-02-11T16:11:47.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-02-11T16:11:47.755 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8" 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8" 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 8" 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-02-11T16:11:47.814 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2023-02-11T16:11:47.815 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-02-11T16:11:47.815 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-02-11T16:11:47.815 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-02-11T16:11:47.815 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2023-02-11T16:11:47.898 INFO:teuthology.dispatcher.supervisor:Running job 7160935 2023-02-11T16:11:47.898 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7160935 --name yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/pool-create-delete} -- /home/teuthworker/archive/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7160935/orig.config.yaml 2023-02-11T16:11:47.906 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7160935 2023-02-11T16:11:47.907 INFO:teuthology.dispatcher.supervisor:Job PID: 30825 2023-02-11T16:11:47.907 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-02-12T04:12:46.943 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-02-12T04:12:52.071 INFO:teuthology.kill:Killing Pids: {30825} 2023-02-12T04:13:05.048 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2023-02-12T04:13:05.048 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-02-12T04:13:05.090 INFO:teuthology.misc:Compressing logs... 2023-02-12T04:13:05.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-02-12T04:13:09.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-02-12T04:13:11.261 INFO:teuthology.kill:No teuthology processes running 2023-02-12T04:13:12.517 INFO:teuthology.kill:Nuking machines: ['smithi040', 'smithi086'] 2023-02-12T04:13:13.649 INFO:teuthology.kill:2023-02-12 04:13:13,648.648 INFO:teuthology.nuke:targets: 2023-02-12T04:13:13.649 INFO:teuthology.kill: smithi040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwXgNsXhredNTwfCBn2LHDvT7xsXruy9KFnST5GuhUXdt98pWu4DfSNvf1+1UD9/DaMwduFuH/XAA0+GhZliD9lyDT607g+18rpmQ+Exk3l1YGaAC2tImqFcPYnj/Is4Qt2naIFUuZgOvpQNk6ipI/Oij4J4nna1sW0ZvMJtKLyE17fOm4b4PRV/djhWOGotavHs/f5g6GCj73jEbs5kAh5ihcpkp6iJYqdIhJa/kkWFMdso9SgbFw/ffqgRLtzopq/s9PGpX1cPstw6W/AtPQxp5+NZFIr7TCQ6YtqT2qWeTeyt4Pgc/ajPxnVgqV8OFf3ZTpeDmmSxd2WiZhfwd141SWaS3ZBua6D976lSD3OQofk5wCwEnSdJr7QYdvjBHRC8RaD+6noHWUk+BngdEwu05WdckKLxUBz6e1XobwBGvqB/1D7N156sQSKYXY+gBqdmkZNHMI8MRO+Uudj/KeUlZnip1aryDhSVoPevNRD2ewrjcD/CCyDOVnj7DJnnU= 2023-02-12T04:13:13.649 INFO:teuthology.kill: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbZFlJ8wrp8LiGrg2PiDCCU1DXTluXmEHxElLhpdLSK6BTEGunQG97QcI4yryme9vRr9UwzMzw8aPY49oCk/UM= 2023-02-12T04:13:13.650 INFO:teuthology.kill:2023-02-12 04:13:13,649.649 INFO:teuthology.nuke:Checking targets against current locks 2023-02-12T04:13:13.705 INFO:teuthology.kill:2023-02-12 04:13:13,705.705 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-02-12T04:13:13.728 INFO:teuthology.kill:2023-02-12 04:13:13,728.728 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-02-12T04:13:13.788 INFO:teuthology.kill:2023-02-12 04:13:13,788.788 INFO:teuthology.orchestra.console:Power off smithi040 2023-02-12T04:13:13.860 INFO:teuthology.kill:2023-02-12 04:13:13,860.860 INFO:teuthology.orchestra.console:Power off smithi086 2023-02-12T04:13:22.328 INFO:teuthology.kill:2023-02-12 04:13:22,327.327 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2023-02-12T04:13:22.469 INFO:teuthology.kill:2023-02-12 04:13:22,469.469 INFO:teuthology.lock.ops:unlocked: smithi086.front.sepia.ceph.com 2023-02-12T04:13:26.445 INFO:teuthology.kill:2023-02-12 04:13:26,445.445 INFO:teuthology.orchestra.console:Power off for smithi040 completed 2023-02-12T04:13:26.591 INFO:teuthology.kill:2023-02-12 04:13:26,591.591 INFO:teuthology.lock.ops:unlocked: smithi040.front.sepia.ceph.com 2023-02-12T04:15:27.010 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-02-12T04:15:27.326 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi040 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-02-08_19:47:33-rados-wip-yuri-testing-2023-02-06-1155-quincy-distro-default-smithi/7161069 2023-02-12T04:15:27.326 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi086 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-02-08_19:47:33-rados-wip-yuri-testing-2023-02-06-1155-quincy-distro-default-smithi/7161069