2022-02-01T23:12:02.973 INFO:root:teuthology version: 1.1.0-0f634dc1
2022-02-01T23:12:03.128 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi043.front.sepia.ceph.com' reimaging
2022-02-01T23:12:03.129 INFO:teuthology.lock.ops:Updating [ubuntu@smithi043.front.sepia.ceph.com]: reset os type and version on server
2022-02-01T23:12:03.129 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server
2022-02-01T23:12:03.148 INFO:teuthology.lock.ops:Node 'ubuntu@smithi043.front.sepia.ceph.com' reimaging is complete
2022-02-01T23:12:03.148 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi070.front.sepia.ceph.com' reimaging
2022-02-01T23:12:03.149 INFO:teuthology.lock.ops:Updating [ubuntu@smithi070.front.sepia.ceph.com]: reset os type and version on server
2022-02-01T23:12:03.149 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server
2022-02-01T23:12:03.168 INFO:teuthology.lock.ops:Node 'ubuntu@smithi070.front.sepia.ceph.com' reimaging is complete
2022-02-01T23:12:03.168 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi189.front.sepia.ceph.com' reimaging
2022-02-01T23:12:03.169 INFO:teuthology.lock.ops:Updating [ubuntu@smithi189.front.sepia.ceph.com]: reset os type and version on server
2022-02-01T23:12:03.169 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server
2022-02-01T23:12:03.188 INFO:teuthology.lock.ops:Node 'ubuntu@smithi189.front.sepia.ceph.com' reimaging is complete
2022-02-01T23:12:03.368 INFO:teuthology.provision.fog.smithi043:Scheduling deploy of ubuntu 20.04
2022-02-01T23:12:03.387 INFO:teuthology.provision.fog.smithi070:Scheduling deploy of ubuntu 20.04
2022-02-01T23:12:03.414 INFO:teuthology.provision.fog.smithi189:Scheduling deploy of ubuntu 20.04
2022-02-01T23:12:03.711 INFO:teuthology.orchestra.console:Power off smithi043
2022-02-01T23:12:03.758 INFO:teuthology.orchestra.console:Power off smithi070
2022-02-01T23:12:03.774 INFO:teuthology.orchestra.console:Power off smithi189
2022-02-01T23:12:12.093 INFO:teuthology.orchestra.console:Power off for smithi043 completed
2022-02-01T23:12:12.122 INFO:teuthology.orchestra.console:Power off for smithi070 completed
2022-02-01T23:12:12.131 INFO:teuthology.orchestra.console:Power off for smithi189 completed
2022-02-01T23:12:12.195 INFO:teuthology.orchestra.console:Power on smithi043
2022-02-01T23:12:12.223 INFO:teuthology.orchestra.console:Power on smithi070
2022-02-01T23:12:12.232 INFO:teuthology.orchestra.console:Power on smithi189
2022-02-01T23:12:20.563 INFO:teuthology.orchestra.console:Power on for smithi043 completed
2022-02-01T23:12:20.583 INFO:teuthology.orchestra.console:Power on for smithi070 completed
2022-02-01T23:12:20.665 INFO:teuthology.provision.fog.smithi043:Waiting for deploy to finish
2022-02-01T23:12:20.684 INFO:teuthology.provision.fog.smithi070:Waiting for deploy to finish
2022-02-01T23:12:20.709 INFO:teuthology.orchestra.console:Power on for smithi189 completed
2022-02-01T23:12:20.810 INFO:teuthology.provision.fog.smithi189:Waiting for deploy to finish
2022-02-01T23:16:58.547 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-01T23:16:59.078 INFO:teuthology.provision.fog.smithi043:Node is ready
2022-02-01T23:16:59.085 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043.front.sepia.ceph.com
2022-02-01T23:16:59.092 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43 smithi043.front.sepia.ceph.com smithi043
2022-02-01T23:16:59.333 INFO:teuthology.provision.fog.smithi043:Deploy complete!
2022-02-01T23:17:22.678 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-01T23:17:23.224 INFO:teuthology.provision.fog.smithi070:Node is ready
2022-02-01T23:17:23.231 INFO:teuthology.orchestra.run.smithi070.stdout:smithi070.front.sepia.ceph.com
2022-02-01T23:17:23.238 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70 smithi070.front.sepia.ceph.com smithi070
2022-02-01T23:17:23.518 INFO:teuthology.provision.fog.smithi070:Deploy complete!
2022-02-01T23:17:46.938 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-01T23:17:47.491 INFO:teuthology.provision.fog.smithi189:Node is ready
2022-02-01T23:17:47.498 INFO:teuthology.orchestra.run.smithi189.stdout:smithi189.front.sepia.ceph.com
2022-02-01T23:17:47.505 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189 smithi189.front.sepia.ceph.com smithi189
2022-02-01T23:17:47.711 INFO:teuthology.provision.fog.smithi189:Deploy complete!
2022-02-01T23:17:47.927 INFO:teuthology.lock.ops:Checking smithi043.front.sepia.ceph.com
2022-02-01T23:17:47.929 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-01T23:17:47.953 INFO:teuthology.lock.ops:Checking smithi070.front.sepia.ceph.com
2022-02-01T23:17:47.954 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-01T23:17:47.973 INFO:teuthology.lock.ops:Checking smithi189.front.sepia.ceph.com
2022-02-01T23:17:47.974 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-01T23:17:48.013 INFO:teuthology.lock.ops:Updating [smithi043.front.sepia.ceph.com]: set os type and version on server
2022-02-01T23:17:48.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com'
2022-02-01T23:17:48.278 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64
2022-02-01T23:17:48.283 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="Ubuntu"
2022-02-01T23:17:48.283 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-01T23:17:48.283 INFO:teuthology.orchestra.run.smithi043.stdout:ID=ubuntu
2022-02-01T23:17:48.284 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE=debian
2022-02-01T23:17:48.284 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-01T23:17:48.284 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="20.04"
2022-02-01T23:17:48.284 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-01T23:17:48.285 INFO:teuthology.orchestra.run.smithi043.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-01T23:17:48.285 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-01T23:17:48.285 INFO:teuthology.orchestra.run.smithi043.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-01T23:17:48.285 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_CODENAME=focal
2022-02-01T23:17:48.286 INFO:teuthology.orchestra.run.smithi043.stdout:UBUNTU_CODENAME=focal
2022-02-01T23:17:48.287 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server
2022-02-01T23:17:48.305 INFO:teuthology.lock.ops:Updating [smithi070.front.sepia.ceph.com]: set os type and version on server
2022-02-01T23:17:48.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com'
2022-02-01T23:17:48.570 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64
2022-02-01T23:17:48.576 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="Ubuntu"
2022-02-01T23:17:48.576 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-01T23:17:48.576 INFO:teuthology.orchestra.run.smithi070.stdout:ID=ubuntu
2022-02-01T23:17:48.576 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE=debian
2022-02-01T23:17:48.577 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-01T23:17:48.577 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="20.04"
2022-02-01T23:17:48.577 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-01T23:17:48.577 INFO:teuthology.orchestra.run.smithi070.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-01T23:17:48.577 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-01T23:17:48.578 INFO:teuthology.orchestra.run.smithi070.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-01T23:17:48.578 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_CODENAME=focal
2022-02-01T23:17:48.578 INFO:teuthology.orchestra.run.smithi070.stdout:UBUNTU_CODENAME=focal
2022-02-01T23:17:48.579 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server
2022-02-01T23:17:48.595 INFO:teuthology.lock.ops:Updating [smithi189.front.sepia.ceph.com]: set os type and version on server
2022-02-01T23:17:48.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com'
2022-02-01T23:17:48.778 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64
2022-02-01T23:17:48.827 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="Ubuntu"
2022-02-01T23:17:48.827 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-01T23:17:48.827 INFO:teuthology.orchestra.run.smithi189.stdout:ID=ubuntu
2022-02-01T23:17:48.828 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE=debian
2022-02-01T23:17:48.828 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-01T23:17:48.828 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="20.04"
2022-02-01T23:17:48.828 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-01T23:17:48.828 INFO:teuthology.orchestra.run.smithi189.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-01T23:17:48.829 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-01T23:17:48.829 INFO:teuthology.orchestra.run.smithi189.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-01T23:17:48.829 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_CODENAME=focal
2022-02-01T23:17:48.829 INFO:teuthology.orchestra.run.smithi189.stdout:UBUNTU_CODENAME=focal
2022-02-01T23:17:48.830 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server
2022-02-01T23:17:48.927 INFO:teuthology.dispatcher.supervisor:Running job 6653657
2022-02-01T23:17:48.927 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2022-02-01_20:31:43-rados-wip-mgr-fixes-pacific-distro-basic-smithi/6653657 --name nojha-2022-02-01_20:31:43-rados-wip-mgr-fixes-pacific-distro-basic-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/nojha-2022-02-01_20:31:43-rados-wip-mgr-fixes-pacific-distro-basic-smithi/6653657/orig.config.yaml
2022-02-01T23:17:48.930 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2022-02-01_20:31:43-rados-wip-mgr-fixes-pacific-distro-basic-smithi/6653657
2022-02-01T23:17:48.931 INFO:teuthology.dispatcher.supervisor:Job PID: 15694
2022-02-01T23:17:48.931 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-02T05:48:18.262 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-02T05:48:18.488 INFO:teuthology.kill:Killing Pids: {15694}
2022-02-02T05:48:19.717 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a']
2022-02-02T05:48:19.718 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x']
2022-02-02T05:48:19.718 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']
2022-02-02T05:48:19.718 INFO:teuthology.misc:Compressing logs...
2022-02-02T05:48:19.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com'
2022-02-02T05:48:19.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com'
2022-02-02T05:48:20.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com'
2022-02-02T05:48:20.355 INFO:teuthology.misc:Compressing logs...
2022-02-02T05:48:21.049 INFO:teuthology.misc:Compressing logs...
2022-02-02T05:53:35.255 INFO:teuthology.kill:No teuthology processes running
2022-02-02T05:53:36.516 INFO:teuthology.kill:Nuking machines: ['smithi043', 'smithi070', 'smithi189']
2022-02-02T05:53:37.472 INFO:teuthology.kill:2022-02-02 05:53:37,472.472 INFO:teuthology.nuke:targets:
2022-02-02T05:53:37.473 INFO:teuthology.kill:  smithi043.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGDtY8QvimSgu33B1eXro75Iv8Y5aykGlBjimg0nW9GlOnm4hfD4kOJmVwwiP8klSaSsoz8qYP9KB5QdebsI+rlWQ3djR4xlSx68uePC/RszleKKV5CLKzcKca2HZAr0g7UxY2LmGBVFpiIPogman4ZgqxNFPHCvjRV2O3J1Hn6XN8VItwtKFJN0j1k0yDnSR3P5oQkn+91a6T2+iuTLfyPC1y81Vc8HcVs5pgvkHHqlUZQTd7agHBFyN9IWdXkEhqZ8WTaER5N7I+j6JQ2KWv1Mui+EPtj4s003NKObQ/o3QmQGkdFHVe2kI8OxQTl+a5qC8l5/xOtXu1T8FrtZuXWwBvuuvifkBFO+aLeUdbFcqJfrJVlPD/CVpo4gpXBsgEEcBKz8NGkCErUbXzCt5jmU0YkFAbL8ihN9LIORSGUP9ylyEyJ/NmGqdIj7QiYbX5Kc1eGg4eLLrk1hZyZZ3E3k1+Pcg+EmDoTlmKKAfEE6xwQLRieG/Y1DtsDS/HfwU=
2022-02-02T05:53:37.474 INFO:teuthology.kill:  smithi070.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs47OnTtp0OJCI9bBnwWpHF1PohbCvx3N/TrBpyOYwsey50QaBQg2Kk4xtdAfw21u0fa+cvWGsZjBpQpv1Y49X2aCkpEJvsTY8yCVItqHMFLv1jXLHauewArQCGu9FigvPrWPsL8fwKKqlgkvfbMWdAi2gJ9EIdZdq9vypAX0cMD+TW68GAWO5K3aFJ+fEAMlo6d7+gMM6jmVRmjX3EvLOH7JUh7cJMOYgl8IUPGMV6qg/r1jYnEHpAlDxXFhHZIWiPh9K5grk5rwJkG0TP9zxDQeum7uwjNQiRDc0V1lFlPrt9UxiEpgaMRV1eBbGh1BNrx/kCPYJxHZKSgOqwE8lM8U5zrBpJq1QazUxBnjQPJITk29NY+3JlzTktib1UT8NLFo3JItb9yY7h6YoGvhesalzq9yB4psFdQ3S4IPVzzjNv/gxbQAvn6emH/fDz2opJd3DVBSCGiors5Hs6HTfG4uq2QpDpMk4n7krdp/NagtYxC8AWJC6M/GFXGUR+2M=
2022-02-02T05:53:37.474 INFO:teuthology.kill:  smithi189.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtz2UPKcZ6nls7uUKWNssWUcfw7onIAMYPoO4BwIgzFuSTZI0ZnmcQh/n3GaAzNhANnzcFt527crkGl3OkCx7ZPNuksPkByv5hxxCDdU6e1t/1otTWSL9wn+D+5FvueQT4BUnpDwpqeIKsgA3rwKL+wnhfkwZRtlgvtm3wn81ThcvX+Uxka965V8xZqQkeCno78hZTm4Oxbl8C9pEEFs5Sdjt5EIuRzYyH1RU++tMpgudpgojzXzIBcdTF0n11AZ6XARURUBSR6dmC1yCCqk3o5z06oZwUWPz5jSvyToGsYUYJ2i5plsK0CdYR1svKoh9vOCgigVPtXBq3K6A907YxXYq4hvag09vsHRhYlgW5nqvwkOOoxCC8pi4oANoj73w4qpFr9nRmGv1kuzdbw8EsJby0zwk3EvFf4T6YBiz7VvtN5jCJLcT8fuj8VH9uyRnEpfxSmg2g36J0asOozez9uVs9KR5CeexsUZ/ivv6bo2aLT6KiW+YNtdeJrnwRL3U=
2022-02-02T05:53:37.500 INFO:teuthology.kill:2022-02-02 05:53:37,500.500 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-02T05:53:37.503 INFO:teuthology.kill:2022-02-02 05:53:37,503.503 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-02T05:53:37.506 INFO:teuthology.kill:2022-02-02 05:53:37,506.506 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-02T05:53:37.585 INFO:teuthology.kill:2022-02-02 05:53:37,585.585 INFO:teuthology.orchestra.console:Power off smithi070
2022-02-02T05:53:37.597 INFO:teuthology.kill:2022-02-02 05:53:37,597.597 INFO:teuthology.orchestra.console:Power off smithi189
2022-02-02T05:53:37.633 INFO:teuthology.kill:2022-02-02 05:53:37,633.633 INFO:teuthology.orchestra.console:Power off smithi043
2022-02-02T05:53:45.975 INFO:teuthology.kill:2022-02-02 05:53:45,975.975 INFO:teuthology.orchestra.console:Power off for smithi189 completed
2022-02-02T05:53:46.122 INFO:teuthology.kill:2022-02-02 05:53:46,121.121 INFO:teuthology.lock.ops:unlocked: smithi189.front.sepia.ceph.com
2022-02-02T05:53:50.080 INFO:teuthology.kill:2022-02-02 05:53:50,080.080 INFO:teuthology.orchestra.console:Power off for smithi070 completed
2022-02-02T05:53:50.114 INFO:teuthology.kill:2022-02-02 05:53:50,114.114 INFO:teuthology.orchestra.console:Power off for smithi043 completed
2022-02-02T05:53:50.228 INFO:teuthology.kill:2022-02-02 05:53:50,227.227 INFO:teuthology.lock.ops:unlocked: smithi070.front.sepia.ceph.com
2022-02-02T05:53:50.264 INFO:teuthology.kill:2022-02-02 05:53:50,264.264 INFO:teuthology.lock.ops:unlocked: smithi043.front.sepia.ceph.com
2022-02-02T05:55:50.594 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-02T05:55:50.719 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi043 but it was locked by another job: /home/teuthworker/archive/pdonnell-2022-02-02_05:50:39-fs:upgrade-wip-pdonnell-testing-20220201.225758-distro-default-smithi/6655148
2022-02-02T05:55:50.719 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi070 but it was locked by another job: /home/teuthworker/archive/pdonnell-2022-02-02_05:50:39-fs:upgrade-wip-pdonnell-testing-20220201.225758-distro-default-smithi/6655148
2022-02-02T05:55:50.719 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi189 but it was locked by another job: /home/teuthworker/archive/pdonnell-2022-02-02_05:50:39-fs:upgrade-wip-pdonnell-testing-20220201.225758-distro-default-smithi/6655147