2022-03-07T22:19:30.777 INFO:root:teuthology version: 1.1.0-eeaeee38
2022-03-07T22:19:30.988 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi073.front.sepia.ceph.com' reimaging
2022-03-07T22:19:30.989 INFO:teuthology.lock.ops:Updating [ubuntu@smithi073.front.sepia.ceph.com]: reset os type and version on server
2022-03-07T22:19:30.989 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server
2022-03-07T22:19:31.009 INFO:teuthology.lock.ops:Node 'ubuntu@smithi073.front.sepia.ceph.com' reimaging is complete
2022-03-07T22:19:31.010 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi106.front.sepia.ceph.com' reimaging
2022-03-07T22:19:31.010 INFO:teuthology.lock.ops:Updating [ubuntu@smithi106.front.sepia.ceph.com]: reset os type and version on server
2022-03-07T22:19:31.010 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server
2022-03-07T22:19:31.031 INFO:teuthology.lock.ops:Node 'ubuntu@smithi106.front.sepia.ceph.com' reimaging is complete
2022-03-07T22:19:31.031 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi112.front.sepia.ceph.com' reimaging
2022-03-07T22:19:31.031 INFO:teuthology.lock.ops:Updating [ubuntu@smithi112.front.sepia.ceph.com]: reset os type and version on server
2022-03-07T22:19:31.032 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server
2022-03-07T22:19:31.050 INFO:teuthology.lock.ops:Node 'ubuntu@smithi112.front.sepia.ceph.com' reimaging is complete
2022-03-07T22:19:31.229 INFO:teuthology.provision.fog.smithi073:Scheduling deploy of ubuntu 20.04
2022-03-07T22:19:31.259 INFO:teuthology.provision.fog.smithi106:Scheduling deploy of ubuntu 20.04
2022-03-07T22:19:31.271 INFO:teuthology.provision.fog.smithi112:Scheduling deploy of ubuntu 20.04
2022-03-07T22:19:31.576 INFO:teuthology.orchestra.console:Power off smithi073
2022-03-07T22:19:31.624 INFO:teuthology.orchestra.console:Power off smithi106
2022-03-07T22:19:31.641 INFO:teuthology.orchestra.console:Power off smithi112
2022-03-07T22:19:44.099 INFO:teuthology.orchestra.console:Power off for smithi073 completed
2022-03-07T22:19:44.109 INFO:teuthology.orchestra.console:Power off for smithi106 completed
2022-03-07T22:19:44.120 INFO:teuthology.orchestra.console:Power off for smithi112 completed
2022-03-07T22:19:44.200 INFO:teuthology.orchestra.console:Power on smithi073
2022-03-07T22:19:44.211 INFO:teuthology.orchestra.console:Power on smithi106
2022-03-07T22:19:44.237 INFO:teuthology.orchestra.console:Power on smithi112
2022-03-07T22:19:48.474 INFO:teuthology.orchestra.console:Power on for smithi106 completed
2022-03-07T22:19:48.575 INFO:teuthology.provision.fog.smithi106:Waiting for deploy to finish
2022-03-07T22:19:52.577 INFO:teuthology.orchestra.console:Power on for smithi073 completed
2022-03-07T22:19:52.680 INFO:teuthology.provision.fog.smithi073:Waiting for deploy to finish
2022-03-07T22:19:52.710 INFO:teuthology.orchestra.console:Power on for smithi112 completed
2022-03-07T22:19:52.811 INFO:teuthology.provision.fog.smithi112:Waiting for deploy to finish
2022-03-07T22:24:44.227 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-07T22:24:44.690 INFO:teuthology.provision.fog.smithi073:Node is ready
2022-03-07T22:24:44.697 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073.front.sepia.ceph.com
2022-03-07T22:24:44.747 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73 smithi073.front.sepia.ceph.com smithi073
2022-03-07T22:24:44.940 INFO:teuthology.provision.fog.smithi073:Deploy complete!
2022-03-07T22:24:49.052 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-07T22:24:49.472 INFO:teuthology.provision.fog.smithi106:Node is ready
2022-03-07T22:24:49.478 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106.front.sepia.ceph.com
2022-03-07T22:24:49.485 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106 smithi106.front.sepia.ceph.com smithi106
2022-03-07T22:24:49.702 INFO:teuthology.provision.fog.smithi106:Deploy complete!
2022-03-07T22:24:53.098 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-07T22:24:53.545 INFO:teuthology.provision.fog.smithi112:Node is ready
2022-03-07T22:24:53.597 INFO:teuthology.orchestra.run.smithi112.stdout:smithi112.front.sepia.ceph.com
2022-03-07T22:24:53.604 INFO:teuthology.orchestra.run.smithi112.stdout:172.21.15.112 smithi112.front.sepia.ceph.com smithi112
2022-03-07T22:24:53.769 INFO:teuthology.provision.fog.smithi112:Deploy complete!
2022-03-07T22:24:54.003 INFO:teuthology.lock.ops:Checking smithi073.front.sepia.ceph.com
2022-03-07T22:24:54.005 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-07T22:24:54.024 INFO:teuthology.lock.ops:Checking smithi106.front.sepia.ceph.com
2022-03-07T22:24:54.024 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-07T22:24:54.043 INFO:teuthology.lock.ops:Checking smithi112.front.sepia.ceph.com
2022-03-07T22:24:54.043 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-07T22:24:54.062 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: set os type and version on server
2022-03-07T22:24:54.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com'
2022-03-07T22:24:54.243 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64
2022-03-07T22:24:54.250 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="Ubuntu"
2022-03-07T22:24:54.250 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-03-07T22:24:54.250 INFO:teuthology.orchestra.run.smithi073.stdout:ID=ubuntu
2022-03-07T22:24:54.251 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE=debian
2022-03-07T22:24:54.251 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-03-07T22:24:54.251 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="20.04"
2022-03-07T22:24:54.251 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://www.ubuntu.com/"
2022-03-07T22:24:54.252 INFO:teuthology.orchestra.run.smithi073.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-03-07T22:24:54.252 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-03-07T22:24:54.252 INFO:teuthology.orchestra.run.smithi073.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-03-07T22:24:54.252 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_CODENAME=focal
2022-03-07T22:24:54.253 INFO:teuthology.orchestra.run.smithi073.stdout:UBUNTU_CODENAME=focal
2022-03-07T22:24:54.255 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server
2022-03-07T22:24:54.278 INFO:teuthology.lock.ops:Updating [smithi106.front.sepia.ceph.com]: set os type and version on server
2022-03-07T22:24:54.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com'
2022-03-07T22:24:54.450 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64
2022-03-07T22:24:54.456 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="Ubuntu"
2022-03-07T22:24:54.457 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-03-07T22:24:54.457 INFO:teuthology.orchestra.run.smithi106.stdout:ID=ubuntu
2022-03-07T22:24:54.457 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE=debian
2022-03-07T22:24:54.457 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-03-07T22:24:54.458 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="20.04"
2022-03-07T22:24:54.458 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://www.ubuntu.com/"
2022-03-07T22:24:54.458 INFO:teuthology.orchestra.run.smithi106.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-03-07T22:24:54.458 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-03-07T22:24:54.459 INFO:teuthology.orchestra.run.smithi106.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-03-07T22:24:54.459 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_CODENAME=focal
2022-03-07T22:24:54.459 INFO:teuthology.orchestra.run.smithi106.stdout:UBUNTU_CODENAME=focal
2022-03-07T22:24:54.459 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server
2022-03-07T22:24:54.477 INFO:teuthology.lock.ops:Updating [smithi112.front.sepia.ceph.com]: set os type and version on server
2022-03-07T22:24:54.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com'
2022-03-07T22:24:54.650 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64
2022-03-07T22:24:54.657 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Ubuntu"
2022-03-07T22:24:54.657 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-03-07T22:24:54.657 INFO:teuthology.orchestra.run.smithi112.stdout:ID=ubuntu
2022-03-07T22:24:54.658 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE=debian
2022-03-07T22:24:54.658 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-03-07T22:24:54.658 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="20.04"
2022-03-07T22:24:54.658 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.ubuntu.com/"
2022-03-07T22:24:54.659 INFO:teuthology.orchestra.run.smithi112.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-03-07T22:24:54.659 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-03-07T22:24:54.659 INFO:teuthology.orchestra.run.smithi112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-03-07T22:24:54.659 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_CODENAME=focal
2022-03-07T22:24:54.660 INFO:teuthology.orchestra.run.smithi112.stdout:UBUNTU_CODENAME=focal
2022-03-07T22:24:54.660 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server
2022-03-07T22:24:54.757 INFO:teuthology.dispatcher.supervisor:Running job 6725242
2022-03-07T22:24:54.757 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-03-07_21:52:24-rados-GBH_safe_shutdown_v2_with_fixes_2-distro-basic-smithi/6725242 --name benhanokh-2022-03-07_21:52:24-rados-GBH_safe_shutdown_v2_with_fixes_2-distro-basic-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/benhanokh-2022-03-07_21:52:24-rados-GBH_safe_shutdown_v2_with_fixes_2-distro-basic-smithi/6725242/orig.config.yaml
2022-03-07T22:24:54.760 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-03-07_21:52:24-rados-GBH_safe_shutdown_v2_with_fixes_2-distro-basic-smithi/6725242
2022-03-07T22:24:54.761 INFO:teuthology.dispatcher.supervisor:Job PID: 31632
2022-03-07T22:24:54.761 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-08T04:55:24.575 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-08T04:55:24.743 INFO:teuthology.kill:Killing Pids: {31632}
2022-03-08T04:55:26.048 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a']
2022-03-08T04:55:26.049 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x']
2022-03-08T04:55:26.049 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']
2022-03-08T04:55:26.049 INFO:teuthology.misc:Compressing logs...
2022-03-08T04:55:26.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com'
2022-03-08T04:55:26.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com'
2022-03-08T04:55:26.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com'
2022-03-08T04:55:26.667 INFO:teuthology.misc:Compressing logs...
2022-03-08T04:55:27.385 INFO:teuthology.misc:Compressing logs...
2022-03-08T04:58:37.540 INFO:teuthology.kill:No teuthology processes running
2022-03-08T04:58:38.650 INFO:teuthology.kill:Nuking machines: ['smithi073', 'smithi106', 'smithi112']
2022-03-08T04:58:39.690 INFO:teuthology.kill:2022-03-08 04:58:39,690.690 INFO:teuthology.nuke:targets:
2022-03-08T04:58:39.691 INFO:teuthology.kill:  smithi073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnyTnFzAMFitX42aoZHRrVcpBoRvtfWXmb1vVHlm6upAwRjIqDnSIdT0wGy+QoV2bfBRbZ2kqUNyFo1Xi6QXzKbRrqZFAYlDqNjfzI1Hm6OVJxZZXHheMBqcrjpD62DtbsPx4num39LVJM15Fe2DoZVRPU83//Bmg/t8nFsBsPt1O7CMqpb6qV5+pS+1YNp4uqncvwsc1//wlvsQu7rDsxzl85iTxMxSqGbwwQ6HWay/ks/uWrumSeTu0wnkXMSJNMftLVyL/Lebmd3ou2OJTlEu17U+ijOdWNrjgt3my0FbqSmo4KouTVFhOURjU808dIgEnJlmgjU7GRGgY5TzaNrkofK5+WwHE9IOPiUEcONmP7mqEn1BXL29/CmLhoBbueZbaHz2J2qAiZNSuXpVNNC2/56u55InFTkuDNHsRN+2SLHMLL2/2Gcy2wqS4qycizwG189JyDaIAUv+0FoeSBb1S6sKGv7XafuZ/Luvmw8FG9E1g+ucqCcoNx/eU71pU=
2022-03-08T04:58:39.691 INFO:teuthology.kill:  smithi106.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClXyEPRHFQ5ZCcihlmAfI4GBfPh6U0Fyx1432xHIauR6xjzzpjiQd83I8vOiAAAf993KiFqbCElRbYwBq/XhCMW/wzsUyNGdaIxIOS12JSwZFcbMvzoDDW8g7rCwWA4qfu8fxg8ahHcYTHNviGbvLayzeAVnrAIYlXYfk7oJiPWCLKeXqOYj/ATy850GvefXBke5ks6CrPxZqhzvwApzbkme7dGOuFFz1CLWtFqZfO5lR+n3suP/L94EnM75tH5Kms9juBfGyY+p743eUj2L29BpYNyyQObEwzQQ8yJTUFaEeheEuKv/4oZdajI+qsMXAt7oM+4nsZxKCfoov6FkBQcSj2lEo9ZJnQ7AGgWIUeEkwSByBHngntHyxdc4SCDOEF3V9ajUmGOeStNYh3wljQuksry6em4wqMYQKWjVqbZXtN3fZ9ugkmZBS5Ni6wTgJTfJTZwSau7vRLm2NbLDMg8Ld1dP3puOC5O1jSb7k8fx3Qquj1ZPl9Jw6SQcrBpcs=
2022-03-08T04:58:39.692 INFO:teuthology.kill:  smithi112.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEgHrAQ8+kcsvGNnIWgTdgblqY/+EnCgDgtqbuu6tapajYj4cMFeEflQ/+zBHh/LZz5v6GISugIoXqax7aAK+KWf6Wft7kIQ2kOZHtYC/63M8FF9nJ2FSlq3yWB95wzkoYYhebUwBHA1QTaoF3wvT6pEj7y+cReiVhFR5NRZ1ut1x2Xykgi/I8UYCJ7jVaAc/j724OyB1xvmKw4nd1i2hK0w6xONTkPr0TgI2IwQBtrb4e/6sR8lWYKMS7HtNtBaFiZp46WBvAUM6nTfexh3ONSUURa33DWduDBf0Msnv6PQRqiqEzrrqv+hjZFRYn6TQQthfeMiZ8luhsDgc5KETB+yIP+e7lf5qkeiDkHutTG+fhyaMVu4n3qJOVujCAQRk1YSp6NABqrplFdKHAR0kMYTkRQ3giZUXqMAA1UWqtuT+jYGK32IF5ghplZvj3jH41xuQ5jc8to/Rvqz5t8AKECxp7wi0hI8qIT4qWDE7eiWd+00ibzUrWvh5khTezaKk=
2022-03-08T04:58:39.722 INFO:teuthology.kill:2022-03-08 04:58:39,722.722 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-08T04:58:39.728 INFO:teuthology.kill:2022-03-08 04:58:39,728.728 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-08T04:58:39.730 INFO:teuthology.kill:2022-03-08 04:58:39,729.729 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-08T04:58:39.822 INFO:teuthology.kill:2022-03-08 04:58:39,822.822 INFO:teuthology.orchestra.console:Power off smithi112
2022-03-08T04:58:39.835 INFO:teuthology.kill:2022-03-08 04:58:39,835.835 INFO:teuthology.orchestra.console:Power off smithi073
2022-03-08T04:58:39.854 INFO:teuthology.kill:2022-03-08 04:58:39,854.854 INFO:teuthology.orchestra.console:Power off smithi106
2022-03-08T04:58:48.240 INFO:teuthology.kill:2022-03-08 04:58:48,240.240 INFO:teuthology.orchestra.console:Power off for smithi112 completed
2022-03-08T04:58:48.388 INFO:teuthology.kill:2022-03-08 04:58:48,388.388 INFO:teuthology.lock.ops:unlocked: smithi112.front.sepia.ceph.com
2022-03-08T04:58:52.344 INFO:teuthology.kill:2022-03-08 04:58:52,343.343 INFO:teuthology.orchestra.console:Power off for smithi073 completed
2022-03-08T04:58:52.361 INFO:teuthology.kill:2022-03-08 04:58:52,361.361 INFO:teuthology.orchestra.console:Power off for smithi106 completed
2022-03-08T04:58:52.486 INFO:teuthology.kill:2022-03-08 04:58:52,486.486 INFO:teuthology.lock.ops:unlocked: smithi073.front.sepia.ceph.com
2022-03-08T04:58:52.505 INFO:teuthology.kill:2022-03-08 04:58:52,505.505 INFO:teuthology.lock.ops:unlocked: smithi106.front.sepia.ceph.com
2022-03-08T05:00:52.814 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-08T05:00:53.008 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi073 but it was locked by another job: /home/teuthworker/archive/cbodley-2022-03-07_22:02:15-rgw-wip-cbodley-testing-distro-default-smithi/6725272
2022-03-08T05:00:53.009 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi106 but it was locked by another job: /home/teuthworker/archive/cbodley-2022-03-07_22:02:15-rgw-wip-cbodley-testing-distro-default-smithi/6725272
2022-03-08T05:00:53.009 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi112 but it was locked by another job: /home/teuthworker/archive/cbodley-2022-03-07_22:02:15-rgw-wip-cbodley-testing-distro-default-smithi/6725273