2022-02-22T18:00:36.155 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-22T18:00:36.303 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging 2022-02-22T18:00:36.304 INFO:teuthology.lock.ops:Updating [ubuntu@smithi090.front.sepia.ceph.com]: reset os type and version on server 2022-02-22T18:00:36.304 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2022-02-22T18:00:36.328 INFO:teuthology.lock.ops:Node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging is complete 2022-02-22T18:00:36.328 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging 2022-02-22T18:00:36.328 INFO:teuthology.lock.ops:Updating [ubuntu@smithi162.front.sepia.ceph.com]: reset os type and version on server 2022-02-22T18:00:36.329 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2022-02-22T18:00:36.348 INFO:teuthology.lock.ops:Node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging is complete 2022-02-22T18:00:36.348 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging 2022-02-22T18:00:36.348 INFO:teuthology.lock.ops:Updating [ubuntu@smithi183.front.sepia.ceph.com]: reset os type and version on server 2022-02-22T18:00:36.349 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2022-02-22T18:00:36.367 INFO:teuthology.lock.ops:Node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging is complete 2022-02-22T18:00:36.546 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of ubuntu 20.04 2022-02-22T18:00:36.562 INFO:teuthology.provision.fog.smithi162:Scheduling deploy of ubuntu 20.04 2022-02-22T18:00:36.595 INFO:teuthology.provision.fog.smithi183:Scheduling deploy of ubuntu 20.04 2022-02-22T18:00:36.883 INFO:teuthology.orchestra.console:Power off smithi090 2022-02-22T18:00:36.897 INFO:teuthology.orchestra.console:Power off smithi162 2022-02-22T18:00:36.920 INFO:teuthology.orchestra.console:Power off smithi183 2022-02-22T18:00:37.029 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2022-02-22T18:00:37.050 INFO:teuthology.orchestra.console:Power off for smithi162 completed 2022-02-22T18:00:37.062 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2022-02-22T18:00:37.130 INFO:teuthology.orchestra.console:Power on smithi090 2022-02-22T18:00:37.156 INFO:teuthology.orchestra.console:Power on smithi162 2022-02-22T18:00:37.163 INFO:teuthology.orchestra.console:Power on smithi183 2022-02-22T18:00:41.385 INFO:teuthology.orchestra.console:Power on for smithi090 completed 2022-02-22T18:00:41.486 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish 2022-02-22T18:00:45.515 INFO:teuthology.orchestra.console:Power on for smithi162 completed 2022-02-22T18:00:45.520 INFO:teuthology.orchestra.console:Power on for smithi183 completed 2022-02-22T18:00:45.617 INFO:teuthology.provision.fog.smithi162:Waiting for deploy to finish 2022-02-22T18:00:45.620 INFO:teuthology.provision.fog.smithi183:Waiting for deploy to finish 2022-02-22T18:05:39.104 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-22T18:05:39.595 INFO:teuthology.provision.fog.smithi183:Node is ready 2022-02-22T18:05:39.601 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183.front.sepia.ceph.com 2022-02-22T18:05:39.609 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.15.183 smithi183.front.sepia.ceph.com smithi183 2022-02-22T18:05:39.797 INFO:teuthology.provision.fog.smithi183:Deploy complete! 2022-02-22T18:05:53.564 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-22T18:05:54.094 INFO:teuthology.provision.fog.smithi162:Node is ready 2022-02-22T18:05:54.101 INFO:teuthology.orchestra.run.smithi162.stdout:smithi162.front.sepia.ceph.com 2022-02-22T18:05:54.149 INFO:teuthology.orchestra.run.smithi162.stdout:172.21.15.162 smithi162.front.sepia.ceph.com smithi162 2022-02-22T18:05:54.324 INFO:teuthology.provision.fog.smithi162:Deploy complete! 2022-02-22T18:06:34.406 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-22T18:06:35.008 INFO:teuthology.provision.fog.smithi090:Node is ready 2022-02-22T18:06:35.014 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com 2022-02-22T18:06:35.062 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090 2022-02-22T18:06:35.269 INFO:teuthology.provision.fog.smithi090:Deploy complete! 2022-02-22T18:06:35.516 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com 2022-02-22T18:06:35.517 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-22T18:06:35.540 INFO:teuthology.lock.ops:Checking smithi162.front.sepia.ceph.com 2022-02-22T18:06:35.540 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-22T18:06:35.557 INFO:teuthology.lock.ops:Checking smithi183.front.sepia.ceph.com 2022-02-22T18:06:35.557 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-22T18:06:35.574 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server 2022-02-22T18:06:35.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2022-02-22T18:06:35.758 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2022-02-22T18:06:35.764 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Ubuntu" 2022-02-22T18:06:35.765 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-22T18:06:35.765 INFO:teuthology.orchestra.run.smithi090.stdout:ID=ubuntu 2022-02-22T18:06:35.765 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE=debian 2022-02-22T18:06:35.765 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-22T18:06:35.766 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="20.04" 2022-02-22T18:06:35.766 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-22T18:06:35.766 INFO:teuthology.orchestra.run.smithi090.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-22T18:06:35.766 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-22T18:06:35.767 INFO:teuthology.orchestra.run.smithi090.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-22T18:06:35.767 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_CODENAME=focal 2022-02-22T18:06:35.767 INFO:teuthology.orchestra.run.smithi090.stdout:UBUNTU_CODENAME=focal 2022-02-22T18:06:35.768 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2022-02-22T18:06:35.788 INFO:teuthology.lock.ops:Updating [smithi162.front.sepia.ceph.com]: set os type and version on server 2022-02-22T18:06:35.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2022-02-22T18:06:35.965 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2022-02-22T18:06:35.971 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="Ubuntu" 2022-02-22T18:06:35.972 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-22T18:06:35.972 INFO:teuthology.orchestra.run.smithi162.stdout:ID=ubuntu 2022-02-22T18:06:35.972 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE=debian 2022-02-22T18:06:35.972 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-22T18:06:35.973 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="20.04" 2022-02-22T18:06:35.973 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-22T18:06:35.973 INFO:teuthology.orchestra.run.smithi162.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-22T18:06:35.973 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-22T18:06:35.974 INFO:teuthology.orchestra.run.smithi162.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-22T18:06:35.974 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_CODENAME=focal 2022-02-22T18:06:35.974 INFO:teuthology.orchestra.run.smithi162.stdout:UBUNTU_CODENAME=focal 2022-02-22T18:06:35.974 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2022-02-22T18:06:35.996 INFO:teuthology.lock.ops:Updating [smithi183.front.sepia.ceph.com]: set os type and version on server 2022-02-22T18:06:35.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2022-02-22T18:06:36.173 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2022-02-22T18:06:36.180 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="Ubuntu" 2022-02-22T18:06:36.180 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-22T18:06:36.180 INFO:teuthology.orchestra.run.smithi183.stdout:ID=ubuntu 2022-02-22T18:06:36.181 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE=debian 2022-02-22T18:06:36.181 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-22T18:06:36.181 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="20.04" 2022-02-22T18:06:36.181 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-22T18:06:36.182 INFO:teuthology.orchestra.run.smithi183.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-22T18:06:36.182 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-22T18:06:36.182 INFO:teuthology.orchestra.run.smithi183.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-22T18:06:36.182 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_CODENAME=focal 2022-02-22T18:06:36.183 INFO:teuthology.orchestra.run.smithi183.stdout:UBUNTU_CODENAME=focal 2022-02-22T18:06:36.183 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2022-02-22T18:06:36.270 INFO:teuthology.dispatcher.supervisor:Running job 6700736 2022-02-22T18:06:36.271 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-02-22_08:31:11-rados-safe_shutdown_v2_FF6-distro-basic-smithi/6700736 --name benhanokh-2022-02-22_08:31:11-rados-safe_shutdown_v2_FF6-distro-basic-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/benhanokh-2022-02-22_08:31:11-rados-safe_shutdown_v2_FF6-distro-basic-smithi/6700736/orig.config.yaml 2022-02-22T18:06:36.274 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-02-22_08:31:11-rados-safe_shutdown_v2_FF6-distro-basic-smithi/6700736 2022-02-22T18:06:36.275 INFO:teuthology.dispatcher.supervisor:Job PID: 32346 2022-02-22T18:06:36.276 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-23T00:37:04.911 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-23T00:37:05.126 INFO:teuthology.kill:Killing Pids: {32346} 2022-02-23T00:37:06.299 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2022-02-23T00:37:06.300 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2022-02-23T00:37:06.300 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2022-02-23T00:37:06.301 INFO:teuthology.misc:Compressing logs... 2022-02-23T00:37:06.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2022-02-23T00:37:06.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2022-02-23T00:37:06.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2022-02-23T00:37:06.968 INFO:teuthology.misc:Compressing logs... 2022-02-23T00:37:07.657 INFO:teuthology.misc:Compressing logs... 2022-02-23T00:39:04.110 INFO:teuthology.kill:No teuthology processes running 2022-02-23T00:39:05.342 INFO:teuthology.kill:Nuking machines: ['smithi090', 'smithi162', 'smithi183'] 2022-02-23T00:39:06.390 INFO:teuthology.kill:2022-02-23 00:39:06,389.389 INFO:teuthology.nuke:targets: 2022-02-23T00:39:06.391 INFO:teuthology.kill: smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2sEDP89n95qdBtE70v/YgEhbPbXd7RMh4qXgrdJsz302Gc4v2T6Q9CyyIzErkXspFQFJqOrM2Os2Ht0dk3YPRwJpI54UJ5kgk2Cewaiq06keCRZdWHg6sEPiTt3Cbyh4K0mSrlODwOyUJu5+L2WtK2PtPMkzszaimL4gEtpDBlaG/8CkiakmaDR1ZWtfcckkFHkf7H/NOPD51zxFr6LXgwB1OIKY5o2haQ0xFp+ajQ+Duoiq6jpgiy1MLjKVa+hZI5oSG9czgmSw1icD+568XAqx+MMdNuph4T+8aTng3mg5Z7ox3MGFygKaRnGivwEHRawbM8JjGdWph0Jsczc1+t28RgSiKJNVbi5Uu19afiDy/iS1ntC1WdBVv92+h1d1moZamv1hOqHv8wQb0inhOXXqHIYXgEhJIqiEyAwZLzW0awZCDfvqEe/nM+xhjJXlNaWhkqbER/gI+vqPSh8RpumAnNnE6a1fHi7kl4ClrV/tK4NoM2LzAlyDQbY7l67k= 2022-02-23T00:39:06.391 INFO:teuthology.kill: smithi162.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+Hf2XzVjFmPRr6YFswPKzno73WnVOzOcR1szTrzeMg+IGfFKnQlG2NErX6Z64epyUo7UAzXxBbo7nsU2n/cy4qVB+3emw3bhBC4Iwy4eWq0rmL4Wk/YiIKr6RIiv6b1dH8yFL2BxaYAsE5JFkw2/Fu3yXF765BjnfhtIuXkhS4Yr+YvSCatL3MLoJ9cwgjdqi0cA6it68/RkgQnojgS+NAypPnv+6YjAXVi5OH7wp0chvALzw32QTWdX4ENhFNtjdCl0NNu1wABK4tB3PGGFYG93XacQ69TNUbLYlUAS99MJE9Dgoi37ZjnT2AqepDOQ+FSMSZxQc/t50FImsaj61hjoR/eCoBYnwC+bWf+WSmbxjxeq8yKJK8otpUX+GOhvMPeYxpX1WjlbNk2QN663u5O+zoOtpxDU3E91MIyuwRYcAzF4RFX5DHFLiIbQkmV/4nyqRj57ngMI0RFk/TW9HiY3TS01H+LiNSU0rgyowzTJiEXyJR8D2/ssgyLWiwlc= 2022-02-23T00:39:06.391 INFO:teuthology.kill: smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC22IjxjA2Pf5BKZGfQGVTpEReVuqv4zTyUdv8QReiNJExREOHe8P7UBGGh3lGD7u8qI5+P6fZA0EXltZMQsBRLMIhmoYmxre+usmyk4QWjEqr2uvHbF0jYxT+H2oexfX0fN8MtRpUFcl9La6mEvhQZNSY7eTdWbYtDA0H4CcPzeG1mTqRR4REgpsSN2BXdixQRkwI0EOB4PKG9K/i6mGf7SYKupjTmwprctpDmyLisqVwS1B2t/zCOmy3UmRQ8Mum5JQnnIvW8BWsPlfpZ/x+MKXwwMFN3fvmEyoVNW2KRFTWcaQoLVYoWpRChfwzS56ybTNVM0Ep2LqQCsD7axJ4E82Ahro1AtWIvwgKAOoZSNjJenVWxulnRahKXD+xV/O6Uuxpm31mGKhseGuCDviJcykbhiRrpWbkmecrR/luxddcgezrnqZahNM07ADdzfmzEFZEhFr8efd+mi/FgJF0f9824jTZZL8Glz/yh7lnNL+f52hx7a6v/SrNkCwDBmpU= 2022-02-23T00:39:06.420 INFO:teuthology.kill:2022-02-23 00:39:06,420.420 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-23T00:39:06.424 INFO:teuthology.kill:2022-02-23 00:39:06,424.424 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-23T00:39:06.426 INFO:teuthology.kill:2022-02-23 00:39:06,426.426 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-23T00:39:06.529 INFO:teuthology.kill:2022-02-23 00:39:06,526.526 INFO:teuthology.orchestra.console:Power off smithi162 2022-02-23T00:39:06.552 INFO:teuthology.kill:2022-02-23 00:39:06,552.552 INFO:teuthology.orchestra.console:Power off smithi183 2022-02-23T00:39:06.573 INFO:teuthology.kill:2022-02-23 00:39:06,572.572 INFO:teuthology.orchestra.console:Power off smithi090 2022-02-23T00:39:14.940 INFO:teuthology.kill:2022-02-23 00:39:14,940.940 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2022-02-23T00:39:14.945 INFO:teuthology.kill:2022-02-23 00:39:14,945.945 INFO:teuthology.orchestra.console:Power off for smithi162 completed 2022-02-23T00:39:15.095 INFO:teuthology.kill:2022-02-23 00:39:15,094.094 INFO:teuthology.lock.ops:unlocked: smithi162.front.sepia.ceph.com 2022-02-23T00:39:15.098 INFO:teuthology.kill:2022-02-23 00:39:15,098.098 INFO:teuthology.lock.ops:unlocked: smithi183.front.sepia.ceph.com 2022-02-23T00:39:19.064 INFO:teuthology.kill:2022-02-23 00:39:19,063.063 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2022-02-23T00:39:19.212 INFO:teuthology.kill:2022-02-23 00:39:19,212.212 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com 2022-02-23T00:41:19.528 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-02-23T00:41:19.714 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi090 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-22_22:10:31-fs-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6701354 2022-02-23T00:41:19.714 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi162 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-22_22:10:31-fs-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6701355 2022-02-23T00:41:19.714 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi183 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-22_22:10:31-fs-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6701354