2022-03-07T22:17:29.783 INFO:root:teuthology version: 1.1.0-eeaeee38
2022-03-07T22:17:30.020 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi134.front.sepia.ceph.com' reimaging
2022-03-07T22:17:30.021 INFO:teuthology.lock.ops:Updating [ubuntu@smithi134.front.sepia.ceph.com]: reset os type and version on server
2022-03-07T22:17:30.022 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server
2022-03-07T22:17:30.040 INFO:teuthology.lock.ops:Node 'ubuntu@smithi134.front.sepia.ceph.com' reimaging is complete
2022-03-07T22:17:30.040 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi141.front.sepia.ceph.com' reimaging
2022-03-07T22:17:30.040 INFO:teuthology.lock.ops:Updating [ubuntu@smithi141.front.sepia.ceph.com]: reset os type and version on server
2022-03-07T22:17:30.041 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server
2022-03-07T22:17:30.061 INFO:teuthology.lock.ops:Node 'ubuntu@smithi141.front.sepia.ceph.com' reimaging is complete
2022-03-07T22:17:30.061 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi148.front.sepia.ceph.com' reimaging
2022-03-07T22:17:30.061 INFO:teuthology.lock.ops:Updating [ubuntu@smithi148.front.sepia.ceph.com]: reset os type and version on server
2022-03-07T22:17:30.062 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server
2022-03-07T22:17:30.085 INFO:teuthology.lock.ops:Node 'ubuntu@smithi148.front.sepia.ceph.com' reimaging is complete
2022-03-07T22:17:30.271 INFO:teuthology.provision.fog.smithi134:Scheduling deploy of ubuntu 20.04
2022-03-07T22:17:30.293 INFO:teuthology.provision.fog.smithi141:Scheduling deploy of ubuntu 20.04
2022-03-07T22:17:30.321 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of ubuntu 20.04
2022-03-07T22:17:30.620 INFO:teuthology.orchestra.console:Power off smithi134
2022-03-07T22:17:30.671 INFO:teuthology.orchestra.console:Power off smithi141
2022-03-07T22:17:30.710 INFO:teuthology.orchestra.console:Power off smithi148
2022-03-07T22:17:39.014 INFO:teuthology.orchestra.console:Power off for smithi134 completed
2022-03-07T22:17:39.070 INFO:teuthology.orchestra.console:Power off for smithi148 completed
2022-03-07T22:17:39.118 INFO:teuthology.orchestra.console:Power on smithi134
2022-03-07T22:17:39.176 INFO:teuthology.orchestra.console:Power on smithi148
2022-03-07T22:17:43.165 INFO:teuthology.orchestra.console:Power off for smithi141 completed
2022-03-07T22:17:43.268 INFO:teuthology.orchestra.console:Power on smithi141
2022-03-07T22:17:47.526 INFO:teuthology.orchestra.console:Power on for smithi134 completed
2022-03-07T22:17:47.564 INFO:teuthology.orchestra.console:Power on for smithi148 completed
2022-03-07T22:17:47.627 INFO:teuthology.provision.fog.smithi134:Waiting for deploy to finish
2022-03-07T22:17:47.665 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish
2022-03-07T22:17:51.639 INFO:teuthology.orchestra.console:Power on for smithi141 completed
2022-03-07T22:17:51.740 INFO:teuthology.provision.fog.smithi141:Waiting for deploy to finish
2022-03-07T22:22:38.795 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-07T22:22:39.282 INFO:teuthology.provision.fog.smithi134:Node is ready
2022-03-07T22:22:39.289 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134.front.sepia.ceph.com
2022-03-07T22:22:39.340 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134 smithi134.front.sepia.ceph.com smithi134
2022-03-07T22:22:39.506 INFO:teuthology.provision.fog.smithi134:Deploy complete!
2022-03-07T22:22:49.042 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-07T22:22:49.447 INFO:teuthology.provision.fog.smithi141:Node is ready
2022-03-07T22:22:49.454 INFO:teuthology.orchestra.run.smithi141.stdout:smithi141.front.sepia.ceph.com
2022-03-07T22:22:49.460 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141 smithi141.front.sepia.ceph.com smithi141
2022-03-07T22:22:49.660 INFO:teuthology.provision.fog.smithi141:Deploy complete!
2022-03-07T22:23:03.123 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-07T22:23:03.526 INFO:teuthology.provision.fog.smithi148:Node is ready
2022-03-07T22:23:03.533 INFO:teuthology.orchestra.run.smithi148.stdout:smithi148.front.sepia.ceph.com
2022-03-07T22:23:03.586 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148 smithi148.front.sepia.ceph.com smithi148
2022-03-07T22:23:03.785 INFO:teuthology.provision.fog.smithi148:Deploy complete!
2022-03-07T22:23:04.008 INFO:teuthology.lock.ops:Checking smithi134.front.sepia.ceph.com
2022-03-07T22:23:04.009 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-07T22:23:04.029 INFO:teuthology.lock.ops:Checking smithi141.front.sepia.ceph.com
2022-03-07T22:23:04.029 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-07T22:23:04.046 INFO:teuthology.lock.ops:Checking smithi148.front.sepia.ceph.com
2022-03-07T22:23:04.047 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-07T22:23:04.064 INFO:teuthology.lock.ops:Updating [smithi134.front.sepia.ceph.com]: set os type and version on server
2022-03-07T22:23:04.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com'
2022-03-07T22:23:04.242 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64
2022-03-07T22:23:04.291 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="Ubuntu"
2022-03-07T22:23:04.291 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-03-07T22:23:04.291 INFO:teuthology.orchestra.run.smithi134.stdout:ID=ubuntu
2022-03-07T22:23:04.292 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE=debian
2022-03-07T22:23:04.292 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-03-07T22:23:04.292 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="20.04"
2022-03-07T22:23:04.292 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://www.ubuntu.com/"
2022-03-07T22:23:04.293 INFO:teuthology.orchestra.run.smithi134.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-03-07T22:23:04.293 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-03-07T22:23:04.293 INFO:teuthology.orchestra.run.smithi134.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-03-07T22:23:04.293 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_CODENAME=focal
2022-03-07T22:23:04.293 INFO:teuthology.orchestra.run.smithi134.stdout:UBUNTU_CODENAME=focal
2022-03-07T22:23:04.294 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server
2022-03-07T22:23:04.319 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: set os type and version on server
2022-03-07T22:23:04.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2022-03-07T22:23:04.502 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64
2022-03-07T22:23:04.508 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="Ubuntu"
2022-03-07T22:23:04.509 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-03-07T22:23:04.509 INFO:teuthology.orchestra.run.smithi141.stdout:ID=ubuntu
2022-03-07T22:23:04.509 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE=debian
2022-03-07T22:23:04.509 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-03-07T22:23:04.510 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="20.04"
2022-03-07T22:23:04.510 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://www.ubuntu.com/"
2022-03-07T22:23:04.510 INFO:teuthology.orchestra.run.smithi141.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-03-07T22:23:04.510 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-03-07T22:23:04.511 INFO:teuthology.orchestra.run.smithi141.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-03-07T22:23:04.511 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_CODENAME=focal
2022-03-07T22:23:04.511 INFO:teuthology.orchestra.run.smithi141.stdout:UBUNTU_CODENAME=focal
2022-03-07T22:23:04.511 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server
2022-03-07T22:23:04.531 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: set os type and version on server
2022-03-07T22:23:04.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com'
2022-03-07T22:23:04.706 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64
2022-03-07T22:23:04.712 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="Ubuntu"
2022-03-07T22:23:04.712 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-03-07T22:23:04.713 INFO:teuthology.orchestra.run.smithi148.stdout:ID=ubuntu
2022-03-07T22:23:04.713 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE=debian
2022-03-07T22:23:04.713 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-03-07T22:23:04.713 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="20.04"
2022-03-07T22:23:04.714 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.ubuntu.com/"
2022-03-07T22:23:04.714 INFO:teuthology.orchestra.run.smithi148.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-03-07T22:23:04.714 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-03-07T22:23:04.715 INFO:teuthology.orchestra.run.smithi148.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-03-07T22:23:04.715 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_CODENAME=focal
2022-03-07T22:23:04.715 INFO:teuthology.orchestra.run.smithi148.stdout:UBUNTU_CODENAME=focal
2022-03-07T22:23:04.715 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server
2022-03-07T22:23:04.810 INFO:teuthology.dispatcher.supervisor:Running job 6725240
2022-03-07T22:23:04.811 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-03-07_21:52:24-rados-GBH_safe_shutdown_v2_with_fixes_2-distro-basic-smithi/6725240 --name benhanokh-2022-03-07_21:52:24-rados-GBH_safe_shutdown_v2_with_fixes_2-distro-basic-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/benhanokh-2022-03-07_21:52:24-rados-GBH_safe_shutdown_v2_with_fixes_2-distro-basic-smithi/6725240/orig.config.yaml
2022-03-07T22:23:04.814 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-03-07_21:52:24-rados-GBH_safe_shutdown_v2_with_fixes_2-distro-basic-smithi/6725240
2022-03-07T22:23:04.815 INFO:teuthology.dispatcher.supervisor:Job PID: 28894
2022-03-07T22:23:04.815 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-08T04:53:34.783 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-08T04:53:34.964 INFO:teuthology.kill:Killing Pids: {28894}
2022-03-08T04:53:36.191 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a']
2022-03-08T04:53:36.191 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x']
2022-03-08T04:53:36.192 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']
2022-03-08T04:53:36.192 INFO:teuthology.misc:Compressing logs...
2022-03-08T04:53:36.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com'
2022-03-08T04:53:36.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2022-03-08T04:53:36.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com'
2022-03-08T04:53:36.951 INFO:teuthology.misc:Compressing logs...
2022-03-08T04:53:37.665 INFO:teuthology.misc:Compressing logs...
2022-03-08T04:56:55.550 INFO:teuthology.kill:No teuthology processes running
2022-03-08T04:56:56.737 INFO:teuthology.kill:Nuking machines: ['smithi134', 'smithi141', 'smithi148']
2022-03-08T04:56:57.754 INFO:teuthology.kill:2022-03-08 04:56:57,754.754 INFO:teuthology.nuke:targets:
2022-03-08T04:56:57.755 INFO:teuthology.kill:  smithi134.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqr69F+RQacc+WO5oy6hhmbxMppsnlkR41zKwwpsaDYPRs3N/eJCM5rf91C1OxN023w8peaupNP8vZCTICUeowJwas0e758I1TcqIiOW4qGp/oyMheEY/Cn+O4io3lqGpGjhGSWhaqWlvZUeiIcWz4hD2sa6PREa/8uavKHQ2KzeI04cjfozGXqmw1b8DG0NK0hq04qGKC2Mhw+2mEaR1YJpPCOKhOhDO2jIaaxahJh3RKBJLyVSyeA+kSk0ZkWU6/4BIWYM5Ig+gvg+LiEkeIwIdwjyZxOau3UEIB2yOye8JwZUzly9t1avM915271fRb1x9uEuZmimkmmTPImx836WYbMC2itfkevA+sMarBrBNGtholCfg9HV4pNTAXsI7CapthBrp61rEgCHjrVXKySjLeidLOTe4qvH846kSKl14TlmVKWNhFN0dAC38AidmpXlWjauOgYYMMkF8CCIeroHZ9zGzSGJzXGLi+9v8Kl+wf7AJMCFy0weEl8VASGT0=
2022-03-08T04:56:57.755 INFO:teuthology.kill:  smithi141.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjtuIzj5E7Fy12kn2tdB9Y7FJ4viGuaPQsKKqaakSfTYaM1RbZgxUDmE6A5LJQoN0lf6Y1qqhNzV2n90IhbBQDoSJhyP4z+Sk3q7AiDFyzyYdSqRuqngPwQIhJ02ocKtq6BBp/Rc9dVJ8+YRikXNNDc1m++9T9JN4GUbfluy76K5RcsDYFSJIj40eZOf57ckFPPZEpuN4mffVcc0MiXvzZff4bpbnNLljsvNKYTygA881oL9mE1A+6vBrsWc4NMSOSF7e4Q7LUAEe9ixaKaSgEee5J3A3EugzFpF1+i/2/UY3yqxjJUmaVjDCl4FPyaFwoqdPmPlh61nYYE3Ws3/K6niKL6zWIEeIca43QHxQh3LTd5ifvZcaphlT8NZ+3dxzoVvSoejVn02zMzYs8F06S8KM3IROZT9o+4ckxmG1FQ6bRPlppnuh8qLc12z3Tey/UW9rHTUM/7NZs4J74WCMuzDF4dTWAL3ziCpcseu08CsBuEns3Y6OxjjL2d9dLuQU=
2022-03-08T04:56:57.755 INFO:teuthology.kill:  smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkldyRggRbOIbvUNVMxyXFvMr0WjMEwbikLUOerFf+EmPC70fws05BIJRjRiaHSVY2Fh93IuZBfMmu+dzlUgzGvVM9lEC1M15+QcVkf+jj5x7RzMLPDQAikSNOEw1xnQ5Lpl/5SeAviYtYbqqHlL4ONn/HTrWF7q33/Wv0QM4C2XL3cPJpmnB3a0AhmpDs6Gh4dWqeDWP9sGHn4qCeurGBJVSbG4hXLTRYLzgD2MPYQhmdPY5nDa6YfI40iDr75pRYUIjM6PX4+GD/3ewGMUbI70N7h2sOlpQ8JCoj/x0CjodNQWwtKPI44v+rPT/AgttpYicpTxWntJtkip+/SxhR8l0Cg1HzJeub+PXk1JoUHAdFK6mjEbYfd2WslR3XJrNljHNBkhtBZHp0xPhI4fIIdktq94xymvaKUXKi9L0YT0FVz229XfF/ESg/RrjBxHJr/tZRo+oQwYV0yvJD+WMEP6yi1x4rpdKyZN45RmvTfAKNILjwRenEZBzUJIAp38k=
2022-03-08T04:56:57.787 INFO:teuthology.kill:2022-03-08 04:56:57,787.787 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-08T04:56:57.792 INFO:teuthology.kill:2022-03-08 04:56:57,792.792 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-08T04:56:57.794 INFO:teuthology.kill:2022-03-08 04:56:57,794.794 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-08T04:56:57.879 INFO:teuthology.kill:2022-03-08 04:56:57,878.878 INFO:teuthology.orchestra.console:Power off smithi141
2022-03-08T04:56:57.895 INFO:teuthology.kill:2022-03-08 04:56:57,895.895 INFO:teuthology.orchestra.console:Power off smithi134
2022-03-08T04:56:57.908 INFO:teuthology.kill:2022-03-08 04:56:57,908.908 INFO:teuthology.orchestra.console:Power off smithi148
2022-03-08T04:57:06.269 INFO:teuthology.kill:2022-03-08 04:57:06,269.269 INFO:teuthology.orchestra.console:Power off for smithi141 completed
2022-03-08T04:57:06.422 INFO:teuthology.kill:2022-03-08 04:57:06,422.422 INFO:teuthology.lock.ops:unlocked: smithi141.front.sepia.ceph.com
2022-03-08T04:57:10.388 INFO:teuthology.kill:2022-03-08 04:57:10,388.388 INFO:teuthology.orchestra.console:Power off for smithi134 completed
2022-03-08T04:57:10.394 INFO:teuthology.kill:2022-03-08 04:57:10,394.394 INFO:teuthology.orchestra.console:Power off for smithi148 completed
2022-03-08T04:57:10.536 INFO:teuthology.kill:2022-03-08 04:57:10,536.536 INFO:teuthology.lock.ops:unlocked: smithi134.front.sepia.ceph.com
2022-03-08T04:57:10.542 INFO:teuthology.kill:2022-03-08 04:57:10,542.542 INFO:teuthology.lock.ops:unlocked: smithi148.front.sepia.ceph.com
2022-03-08T04:59:10.872 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-08T04:59:11.000 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi134 but it was locked by another job: /home/teuthworker/archive/cbodley-2022-03-07_22:02:15-rgw-wip-cbodley-testing-distro-default-smithi/6725261
2022-03-08T04:59:11.001 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi141 but it was locked by another job: /home/teuthworker/archive/cbodley-2022-03-07_22:02:15-rgw-wip-cbodley-testing-distro-default-smithi/6725262
2022-03-08T04:59:11.001 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi148 but it was locked by another job: /home/teuthworker/archive/cbodley-2022-03-07_22:02:15-rgw-wip-cbodley-testing-distro-default-smithi/6725261