2022-04-29T06:25:25.128 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-04-29T06:25:25.374 INFO:teuthology.lock.ops:Start node 'smithi062.front.sepia.ceph.com' reimaging 2022-04-29T06:25:25.376 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: reset os type and version on server 2022-04-29T06:25:25.377 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2022-04-29T06:25:25.409 INFO:teuthology.lock.ops:Node 'smithi062.front.sepia.ceph.com' reimaging is complete 2022-04-29T06:25:25.417 INFO:teuthology.lock.ops:Start node 'smithi085.front.sepia.ceph.com' reimaging 2022-04-29T06:25:25.417 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: reset os type and version on server 2022-04-29T06:25:25.418 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2022-04-29T06:25:25.440 INFO:teuthology.lock.ops:Node 'smithi085.front.sepia.ceph.com' reimaging is complete 2022-04-29T06:25:25.440 INFO:teuthology.lock.ops:Start node 'smithi109.front.sepia.ceph.com' reimaging 2022-04-29T06:25:25.440 INFO:teuthology.lock.ops:Updating [smithi109.front.sepia.ceph.com]: reset os type and version on server 2022-04-29T06:25:25.440 INFO:teuthology.lock.ops:Updating smithi109.front.sepia.ceph.com on lock server 2022-04-29T06:25:25.461 INFO:teuthology.lock.ops:Node 'smithi109.front.sepia.ceph.com' reimaging is complete 2022-04-29T06:25:25.682 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of ubuntu 18.04 2022-04-29T06:25:25.694 INFO:teuthology.provision.fog.smithi109:Scheduling deploy of ubuntu 18.04 2022-04-29T06:25:25.705 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of ubuntu 18.04 2022-04-29T06:25:26.096 INFO:teuthology.orchestra.console:Power off smithi062 2022-04-29T06:25:26.147 INFO:teuthology.orchestra.console:Power off smithi109 2022-04-29T06:25:26.212 INFO:teuthology.orchestra.console:Power off smithi085 2022-04-29T06:25:34.488 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2022-04-29T06:25:34.590 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2022-04-29T06:25:34.591 INFO:teuthology.orchestra.console:Power on smithi062 2022-04-29T06:25:34.691 INFO:teuthology.orchestra.console:Power on smithi085 2022-04-29T06:25:38.630 INFO:teuthology.orchestra.console:Power off for smithi109 completed 2022-04-29T06:25:38.731 INFO:teuthology.orchestra.console:Power on smithi109 2022-04-29T06:25:42.968 INFO:teuthology.orchestra.console:Power on for smithi062 completed 2022-04-29T06:25:42.986 INFO:teuthology.orchestra.console:Power on for smithi109 completed 2022-04-29T06:25:43.061 INFO:teuthology.orchestra.console:Power on for smithi085 completed 2022-04-29T06:25:43.084 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish 2022-04-29T06:25:43.089 INFO:teuthology.provision.fog.smithi109:Waiting for deploy to finish 2022-04-29T06:25:43.162 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish 2022-04-29T06:30:45.453 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-29T06:30:45.493 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-29T06:30:45.746 INFO:teuthology.provision.fog.smithi062:Node is ready 2022-04-29T06:30:45.771 INFO:teuthology.provision.fog.smithi085:Node is ready 2022-04-29T06:30:45.796 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com 2022-04-29T06:30:45.830 INFO:teuthology.orchestra.run.smithi085.stdout:smithi085.front.sepia.ceph.com 2022-04-29T06:30:45.868 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062 2022-04-29T06:30:45.879 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.85 smithi085.front.sepia.ceph.com smithi085 2022-04-29T06:30:46.247 INFO:teuthology.provision.fog.smithi062:Deploy complete! 2022-04-29T06:30:46.300 INFO:teuthology.provision.fog.smithi085:Deploy complete! 2022-04-29T06:30:54.362 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-29T06:30:54.728 INFO:teuthology.provision.fog.smithi109:Node is ready 2022-04-29T06:30:54.782 INFO:teuthology.orchestra.run.smithi109.stdout:smithi109.front.sepia.ceph.com 2022-04-29T06:30:54.851 INFO:teuthology.orchestra.run.smithi109.stdout:172.21.15.109 smithi109.front.sepia.ceph.com smithi109 2022-04-29T06:30:55.214 INFO:teuthology.provision.fog.smithi109:Deploy complete! 2022-04-29T06:30:55.487 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com 2022-04-29T06:30:55.487 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-29T06:30:55.508 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com 2022-04-29T06:30:55.509 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-29T06:30:55.536 INFO:teuthology.lock.ops:Checking smithi109.front.sepia.ceph.com 2022-04-29T06:30:55.537 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-29T06:30:55.572 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server 2022-04-29T06:30:55.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2022-04-29T06:30:55.888 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2022-04-29T06:30:55.938 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Ubuntu" 2022-04-29T06:30:55.939 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2022-04-29T06:30:55.939 INFO:teuthology.orchestra.run.smithi062.stdout:ID=ubuntu 2022-04-29T06:30:55.939 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE=debian 2022-04-29T06:30:55.939 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2022-04-29T06:30:55.939 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="18.04" 2022-04-29T06:30:55.939 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-29T06:30:55.940 INFO:teuthology.orchestra.run.smithi062.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-29T06:30:55.940 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-29T06:30:55.940 INFO:teuthology.orchestra.run.smithi062.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-29T06:30:55.940 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_CODENAME=bionic 2022-04-29T06:30:55.940 INFO:teuthology.orchestra.run.smithi062.stdout:UBUNTU_CODENAME=bionic 2022-04-29T06:30:55.941 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2022-04-29T06:30:55.968 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server 2022-04-29T06:30:55.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2022-04-29T06:30:56.342 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2022-04-29T06:30:56.389 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="Ubuntu" 2022-04-29T06:30:56.391 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2022-04-29T06:30:56.391 INFO:teuthology.orchestra.run.smithi085.stdout:ID=ubuntu 2022-04-29T06:30:56.391 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE=debian 2022-04-29T06:30:56.391 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2022-04-29T06:30:56.391 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="18.04" 2022-04-29T06:30:56.391 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-29T06:30:56.391 INFO:teuthology.orchestra.run.smithi085.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-29T06:30:56.392 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-29T06:30:56.392 INFO:teuthology.orchestra.run.smithi085.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-29T06:30:56.401 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_CODENAME=bionic 2022-04-29T06:30:56.401 INFO:teuthology.orchestra.run.smithi085.stdout:UBUNTU_CODENAME=bionic 2022-04-29T06:30:56.402 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2022-04-29T06:30:56.420 INFO:teuthology.lock.ops:Updating [smithi109.front.sepia.ceph.com]: set os type and version on server 2022-04-29T06:30:56.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi109.front.sepia.ceph.com' 2022-04-29T06:30:56.749 INFO:teuthology.orchestra.run.smithi109.stdout:x86_64 2022-04-29T06:30:56.798 INFO:teuthology.orchestra.run.smithi109.stdout:NAME="Ubuntu" 2022-04-29T06:30:56.798 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2022-04-29T06:30:56.804 INFO:teuthology.orchestra.run.smithi109.stdout:ID=ubuntu 2022-04-29T06:30:56.804 INFO:teuthology.orchestra.run.smithi109.stdout:ID_LIKE=debian 2022-04-29T06:30:56.804 INFO:teuthology.orchestra.run.smithi109.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2022-04-29T06:30:56.805 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION_ID="18.04" 2022-04-29T06:30:56.805 INFO:teuthology.orchestra.run.smithi109.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-29T06:30:56.805 INFO:teuthology.orchestra.run.smithi109.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-29T06:30:56.805 INFO:teuthology.orchestra.run.smithi109.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-29T06:30:56.805 INFO:teuthology.orchestra.run.smithi109.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-29T06:30:56.805 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION_CODENAME=bionic 2022-04-29T06:30:56.805 INFO:teuthology.orchestra.run.smithi109.stdout:UBUNTU_CODENAME=bionic 2022-04-29T06:30:56.806 INFO:teuthology.lock.ops:Updating smithi109.front.sepia.ceph.com on lock server 2022-04-29T06:30:56.932 INFO:teuthology.dispatcher.supervisor:Running job 6810189 2022-04-29T06:30:56.932 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_135488acc76a85490d37201ec506706cc4ab2d62/virtualenv/bin/teuthology -v --owner scheduled_lflores@teuthology --archive /home/teuthworker/archive/lflores-2022-04-27_17:53:15-rados:thrash-erasure-code-big-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6810189 --name lflores-2022-04-27_17:53:15-rados:thrash-erasure-code-big-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi --description rados:thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_18.04} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/lflores-2022-04-27_17:53:15-rados:thrash-erasure-code-big-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6810189/orig.config.yaml 2022-04-29T06:30:56.949 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/lflores-2022-04-27_17:53:15-rados:thrash-erasure-code-big-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6810189 2022-04-29T06:30:56.957 INFO:teuthology.dispatcher.supervisor:Job PID: 1005 2022-04-29T06:30:56.958 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-04-29T13:01:27.507 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-04-29T13:01:27.675 INFO:teuthology.kill:Killing Pids: {1005} 2022-04-29T13:01:30.106 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2022-04-29T13:01:30.107 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2022-04-29T13:01:30.107 INFO:teuthology.task.internal:roles: ubuntu@smithi109.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2022-04-29T13:01:30.107 INFO:teuthology.misc:Compressing logs... 2022-04-29T13:01:30.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2022-04-29T13:01:30.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2022-04-29T13:01:30.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi109.front.sepia.ceph.com' 2022-04-29T13:01:31.554 INFO:teuthology.kill:No teuthology processes running 2022-04-29T13:01:32.925 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi085', 'smithi109'] 2022-04-29T13:01:34.199 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_135488acc76a85490d37201ec506706cc4ab2d62/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-04-29T13:01:34.199 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-04-29T13:01:34.199 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_135488acc76a85490d37201ec506706cc4ab2d62/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated 2022-04-29T13:01:34.199 INFO:teuthology.kill: "class": algorithms.Blowfish, 2022-04-29T13:01:34.200 INFO:teuthology.kill:2022-04-29 13:01:34,198.198 INFO:teuthology.nuke:targets: 2022-04-29T13:01:34.200 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfhDlfWamooWbiaoDeNWCsZcQTqszwTxdBHkOX/WRiPh5/cyDsNzSYkzDBPEKbaNTro+D5JSdRwtzvdc6P7+MVi3Rek+MHqwy2ydTyOCmPvpIgFeEuIliibfK2ftxNuRiLLtV8xjrv7bNgBcDCaDx/+wnhuS2m08wsvQ+n1oUoZw6yKccDXt9BRHQqb72smoonMwLKYGzJLbUZVCFEICbB88FetZcns3Y28Cw40B8UJQV2p6gsRabPyFGKIjyIuOLKh9Z9oz6nsFEfMbVGbtP+oRYBtPfeKS/OCgCZdoXYDk3jhTLFeMFFfTlS0QGT8FxDLg5dboQJLfzRkuyXs6mJ 2022-04-29T13:01:34.200 INFO:teuthology.kill: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw8j4cIzDspXr7IdbEdX9nFkCssIY2H9WYkVAsqCS33GpKR3YlKs2zQMj3tKs6RH9k3FzR+uUggPsSmBAo/kV0ljnfmN8xkaOvFT/TEYJDQbrXHhuK1/Qy7vq98oDU848jOIcPgO0AWTc0quhDxCVFWBeHq6lqvh4jxq7jK8Dt6J//gd3gpIcsKOqtMaEAlmVNbH+aJ19tt8v49nUGGZzQiJcl2nSWs3FcXQaMNLJnE2RHPOO1QiT4OFNs+u2VFZlpmIpLdEnvz+08mbDmlA5bnFumvtNGurjYPuZRu05KVq8ar5NcfxeIARu26d5SFeb/VkYQYODUlpy7/I6vnq1b 2022-04-29T13:01:34.200 INFO:teuthology.kill: smithi109.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5KKGF6LDE5mjDBQVySMob72RxAwCY1LRiKYz3x98Zy96U0aLHneiJw4w4XQ9id0wu305P0uG+ovaexcQgFW2ZVFAnMhccxNSzc2b7z/WX8SzkIU1bgog9wu5rPF2XPntWab2T6chzlLWiDMsmeJVXha/iEusImVUXEnveCfm7ZstMo+CdbZBT0+MTM6gjBvipb3c9Hk2lKLVBURPQfbvZtNzS7LfhkmFDjzFo8jAG/3Rtwjjzggqw841y71bfpfrQ18DPcYON250C7iEIroIUvW1AawExwuK7gAkijOT6idjXAshCXpwCvGtiOXpThwPrasjO6O+6r3ahUVpIdxst 2022-04-29T13:01:34.230 INFO:teuthology.kill:2022-04-29 13:01:34,230.230 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-29T13:01:34.235 INFO:teuthology.kill:2022-04-29 13:01:34,234.234 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-29T13:01:34.236 INFO:teuthology.kill:2022-04-29 13:01:34,236.236 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-29T13:01:34.321 INFO:teuthology.kill:2022-04-29 13:01:34,320.320 INFO:teuthology.orchestra.console:Power off smithi085 2022-04-29T13:01:34.336 INFO:teuthology.kill:2022-04-29 13:01:34,335.335 INFO:teuthology.orchestra.console:Power off smithi109 2022-04-29T13:01:34.349 INFO:teuthology.kill:2022-04-29 13:01:34,349.349 INFO:teuthology.orchestra.console:Power off smithi062 2022-04-29T13:01:42.718 INFO:teuthology.kill:2022-04-29 13:01:42,718.718 INFO:teuthology.orchestra.console:Power off for smithi109 completed 2022-04-29T13:01:42.865 INFO:teuthology.kill:2022-04-29 13:01:42,864.864 INFO:teuthology.lock.ops:unlocked: smithi109.front.sepia.ceph.com 2022-04-29T13:01:46.820 INFO:teuthology.kill:2022-04-29 13:01:46,820.820 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2022-04-29T13:01:46.836 INFO:teuthology.kill:2022-04-29 13:01:46,835.835 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2022-04-29T13:01:46.977 INFO:teuthology.kill:2022-04-29 13:01:46,976.976 INFO:teuthology.lock.ops:unlocked: smithi085.front.sepia.ceph.com 2022-04-29T13:01:46.981 INFO:teuthology.kill:2022-04-29 13:01:46,981.981 INFO:teuthology.lock.ops:unlocked: smithi062.front.sepia.ceph.com 2022-04-29T13:03:47.328 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-04-29T13:03:47.465 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi062 but it was locked by another job: /home/teuthworker/archive/jcollin-2022-04-29_12:48:29-fs-wip-F40633-dump-recent-extraordinary-events-distro-default-smithi/6813250 2022-04-29T13:03:47.465 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi085 but it was locked by another job: /home/teuthworker/archive/jcollin-2022-04-29_12:48:29-fs-wip-F40633-dump-recent-extraordinary-events-distro-default-smithi/6813250 2022-04-29T13:03:47.466 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi109 but it was locked by another job: /home/teuthworker/archive/jcollin-2022-04-29_12:48:29-fs-wip-F40633-dump-recent-extraordinary-events-distro-default-smithi/6813250