2022-05-10T18:10:55.202 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-10T18:10:55.334 INFO:teuthology.lock.ops:Start node 'smithi062.front.sepia.ceph.com' reimaging 2022-05-10T18:10:55.336 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: reset os type and version on server 2022-05-10T18:10:55.336 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2022-05-10T18:10:55.355 INFO:teuthology.lock.ops:Node 'smithi062.front.sepia.ceph.com' reimaging is complete 2022-05-10T18:10:55.356 INFO:teuthology.lock.ops:Start node 'smithi139.front.sepia.ceph.com' reimaging 2022-05-10T18:10:55.356 INFO:teuthology.lock.ops:Updating [smithi139.front.sepia.ceph.com]: reset os type and version on server 2022-05-10T18:10:55.357 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2022-05-10T18:10:55.377 INFO:teuthology.lock.ops:Node 'smithi139.front.sepia.ceph.com' reimaging is complete 2022-05-10T18:10:55.567 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of ubuntu 20.04 2022-05-10T18:10:55.576 INFO:teuthology.provision.fog.smithi139:Scheduling deploy of ubuntu 20.04 2022-05-10T18:10:55.873 INFO:teuthology.orchestra.console:Power off smithi062 2022-05-10T18:10:55.888 INFO:teuthology.orchestra.console:Power off smithi139 2022-05-10T18:11:08.351 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2022-05-10T18:11:08.359 INFO:teuthology.orchestra.console:Power off for smithi139 completed 2022-05-10T18:11:08.454 INFO:teuthology.orchestra.console:Power on smithi062 2022-05-10T18:11:08.460 INFO:teuthology.orchestra.console:Power on smithi139 2022-05-10T18:11:16.816 INFO:teuthology.orchestra.console:Power on for smithi062 completed 2022-05-10T18:11:16.821 INFO:teuthology.orchestra.console:Power on for smithi139 completed 2022-05-10T18:11:16.917 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish 2022-05-10T18:11:16.921 INFO:teuthology.provision.fog.smithi139:Waiting for deploy to finish 2022-05-10T18:16:05.943 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-10T18:16:06.569 INFO:teuthology.provision.fog.smithi062:Node is ready 2022-05-10T18:16:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com 2022-05-10T18:16:06.585 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062 2022-05-10T18:16:06.810 INFO:teuthology.provision.fog.smithi062:Deploy complete! 2022-05-10T18:16:15.047 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-10T18:16:15.557 INFO:teuthology.provision.fog.smithi139:Node is ready 2022-05-10T18:16:15.607 INFO:teuthology.orchestra.run.smithi139.stdout:smithi139.front.sepia.ceph.com 2022-05-10T18:16:15.614 INFO:teuthology.orchestra.run.smithi139.stdout:172.21.15.139 smithi139.front.sepia.ceph.com smithi139 2022-05-10T18:16:15.777 INFO:teuthology.provision.fog.smithi139:Deploy complete! 2022-05-10T18:16:15.966 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com 2022-05-10T18:16:15.967 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-10T18:16:15.986 INFO:teuthology.lock.ops:Checking smithi139.front.sepia.ceph.com 2022-05-10T18:16:15.987 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-10T18:16:16.005 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server 2022-05-10T18:16:16.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2022-05-10T18:16:16.230 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2022-05-10T18:16:16.236 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Ubuntu" 2022-05-10T18:16:16.236 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-05-10T18:16:16.237 INFO:teuthology.orchestra.run.smithi062.stdout:ID=ubuntu 2022-05-10T18:16:16.237 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE=debian 2022-05-10T18:16:16.237 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-05-10T18:16:16.237 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="20.04" 2022-05-10T18:16:16.238 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.ubuntu.com/" 2022-05-10T18:16:16.238 INFO:teuthology.orchestra.run.smithi062.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-05-10T18:16:16.238 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-05-10T18:16:16.238 INFO:teuthology.orchestra.run.smithi062.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-05-10T18:16:16.239 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_CODENAME=focal 2022-05-10T18:16:16.239 INFO:teuthology.orchestra.run.smithi062.stdout:UBUNTU_CODENAME=focal 2022-05-10T18:16:16.240 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2022-05-10T18:16:16.258 INFO:teuthology.lock.ops:Updating [smithi139.front.sepia.ceph.com]: set os type and version on server 2022-05-10T18:16:16.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2022-05-10T18:16:16.523 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2022-05-10T18:16:16.530 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="Ubuntu" 2022-05-10T18:16:16.530 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-05-10T18:16:16.530 INFO:teuthology.orchestra.run.smithi139.stdout:ID=ubuntu 2022-05-10T18:16:16.531 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE=debian 2022-05-10T18:16:16.531 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-05-10T18:16:16.531 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="20.04" 2022-05-10T18:16:16.531 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://www.ubuntu.com/" 2022-05-10T18:16:16.531 INFO:teuthology.orchestra.run.smithi139.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-05-10T18:16:16.532 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-05-10T18:16:16.532 INFO:teuthology.orchestra.run.smithi139.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-05-10T18:16:16.532 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_CODENAME=focal 2022-05-10T18:16:16.532 INFO:teuthology.orchestra.run.smithi139.stdout:UBUNTU_CODENAME=focal 2022-05-10T18:16:16.533 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2022-05-10T18:16:16.634 INFO:teuthology.dispatcher.supervisor:Running job 6829902 2022-05-10T18:16:16.634 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-05-10_04:17:03-fs-pacific-distro-default-smithi/6829902 --name teuthology-2022-05-10_04:17:03-fs-pacific-distro-default-smithi --description fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} -- /home/teuthworker/archive/teuthology-2022-05-10_04:17:03-fs-pacific-distro-default-smithi/6829902/orig.config.yaml 2022-05-10T18:16:16.637 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-05-10_04:17:03-fs-pacific-distro-default-smithi/6829902 2022-05-10T18:16:16.638 INFO:teuthology.dispatcher.supervisor:Job PID: 26413 2022-05-10T18:16:16.639 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-05-11T00:46:41.139 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-11T00:46:41.358 INFO:teuthology.kill:Killing Pids: {26413} 2022-05-11T00:46:42.652 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-05-11T00:46:42.653 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-05-11T00:46:42.653 INFO:teuthology.misc:Compressing logs... 2022-05-11T00:46:42.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2022-05-11T00:46:43.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2022-05-11T00:46:43.333 INFO:teuthology.misc:Compressing logs... 2022-05-11T00:46:44.342 INFO:teuthology.misc:Compressing logs... 2022-05-11T00:46:45.603 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2022-05-11T00:46:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2022-05-11T00:46:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/ceph-mds.a.log: file size changed while zipping 2022-05-11T00:46:46.010 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/ceph-mds.c.log: file size changed while zipping 2022-05-11T00:46:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2022-05-11T00:46:48.143 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2022-05-11T00:46:48.367 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2022-05-11T00:46:50.084 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2022-05-11T00:46:51.115 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2022-05-11T00:46:51.965 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2022-05-11T00:46:53.060 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping 2022-05-11T00:47:00.496 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2022-05-11T00:47:01.794 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2022-05-11T00:47:02.845 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2022-05-11T00:47:03.032 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2022-05-11T01:53:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:tar: .: file changed as we read it 2022-05-11T03:08:50.992 INFO:teuthology.kill:No teuthology processes running 2022-05-11T03:08:52.246 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi139'] 2022-05-11T03:08:53.292 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-05-11T03:08:53.293 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-05-11T03:08:53.293 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b018fa018d3f885cbc018f839b42536477653f84/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated 2022-05-11T03:08:53.293 INFO:teuthology.kill: "class": algorithms.Blowfish, 2022-05-11T03:08:53.293 INFO:teuthology.kill:2022-05-11 03:08:53,291.291 INFO:teuthology.nuke:targets: 2022-05-11T03:08:53.294 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5hj3l9tThfFTr3RKfWtvdmX3pYrueADTqf/4s7JqgzKFG/1q+fDlW6Mqv3pID/9qrHGtaL8JXeKIYC7Ns2tjUe+cTIQ7MzmLO3MsId0ODBL3fHYgiTi+D+pQbYxfPItAuDf0bM3/wNNhOszyEFHqAu1U8PLAkfApS05pltSBRepOMWNgbdy2KgxL1G8Q+wTPQ4YUBC5cLR0UHxLRy0a1ecBI6y4Za6N/IkjAdrn3XQnKshjjzBov+MV+ybLF8Z5OP7fQOgKgXpaCgI9+ucig5azpo0WD/eUyyjSXA7VfWCTE+45pbZig0GPd/R2JJUilZt5Lb6SEWOu8jZlvtWwDQ6ocZc+V1sz6ywgvUAnPWbdjgFf2TjPanJrv1CsKKwVwxEuWkawVEd73KSDevlDUiN7pzzL4ZNY+k1l3z/WeZBjuK8NaRwL4Qp7I5JIaqFG1LNgwHRLfsjTF7fQPjkaXkUv6tmzdYIRsB+VnMCQR126cWOvUvkTJuI5mHB1weNs8= 2022-05-11T03:08:53.294 INFO:teuthology.kill: smithi139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcxwsXfX5QtZCPO2EIiErJKa0t9o6D3tu//L62Tbc2LlISqo+JOtllsnmg1y6caXxBeLNDwj3prX16dTgRjUFm69zMxhq7EQa/KJGkgkd5j6X96woZY5ls4/jchys2yzlQUX8cWRE+EoRzEJp7emang5xmiC+aVhLiXX1keoc7Ixl9JppEhyZmc4FQYJsfbiZn4UyaiCIPDub8wIN7LZ1bLOzzvYp+A+oqpvrVQ95pKHnpwZ9Tyk0CBUGZZFsREQh6DTfDnOHQyfj7x0wcH0a0mNBkhQe5k7DbQf40E/23bokvsKsifOa++k/2SZt7jKTaOBLYFs/5aPNI/fLarhrfxHpzGYVNjIeQ49s3y+uepfz7OVmYghM/WMXEdo4QleKjVGW/pWaT1Scc34812WdGty4/osu6f4MbJxBlN7Uzd7htosZbpG5CNTb5gC9TBKwokoU3cEhfrt2X5uzUrzJXwMc6p1ln+LvCtWlAjmKC2efysgs+3riVPQC10+qWFzU= 2022-05-11T03:08:53.320 INFO:teuthology.kill:2022-05-11 03:08:53,320.320 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-11T03:08:53.323 INFO:teuthology.kill:2022-05-11 03:08:53,323.323 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-11T03:08:53.395 INFO:teuthology.kill:2022-05-11 03:08:53,395.395 INFO:teuthology.orchestra.console:Power off smithi062 2022-05-11T03:08:53.407 INFO:teuthology.kill:2022-05-11 03:08:53,406.406 INFO:teuthology.orchestra.console:Power off smithi139 2022-05-11T03:09:01.766 INFO:teuthology.kill:2022-05-11 03:09:01,766.766 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2022-05-11T03:09:01.918 INFO:teuthology.kill:2022-05-11 03:09:01,917.917 INFO:teuthology.lock.ops:unlocked: smithi062.front.sepia.ceph.com 2022-05-11T03:09:05.883 INFO:teuthology.kill:2022-05-11 03:09:05,882.882 INFO:teuthology.orchestra.console:Power off for smithi139 completed 2022-05-11T03:09:06.027 INFO:teuthology.kill:2022-05-11 03:09:06,027.027 INFO:teuthology.lock.ops:unlocked: smithi139.front.sepia.ceph.com 2022-05-11T03:11:06.349 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15