2022-04-26T13:46:56.108 INFO:root:teuthology version: 1.1.1.dev317+g788cfdd8
2022-04-26T13:46:56.269 INFO:teuthology.lock.ops:Start node 'smithi119.front.sepia.ceph.com' reimaging
2022-04-26T13:46:56.270 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: reset os type and version on server
2022-04-26T13:46:56.270 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server
2022-04-26T13:46:56.290 INFO:teuthology.lock.ops:Node 'smithi119.front.sepia.ceph.com' reimaging is complete
2022-04-26T13:46:56.290 INFO:teuthology.lock.ops:Start node 'smithi196.front.sepia.ceph.com' reimaging
2022-04-26T13:46:56.291 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: reset os type and version on server
2022-04-26T13:46:56.291 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server
2022-04-26T13:46:56.309 INFO:teuthology.lock.ops:Node 'smithi196.front.sepia.ceph.com' reimaging is complete
2022-04-26T13:46:56.503 INFO:teuthology.provision.fog.smithi119:Scheduling deploy of centos 8.stream
2022-04-26T13:46:56.524 INFO:teuthology.provision.fog.smithi196:Scheduling deploy of centos 8.stream
2022-04-26T13:46:56.867 INFO:teuthology.orchestra.console:Power off smithi119
2022-04-26T13:46:56.894 INFO:teuthology.orchestra.console:Power off smithi196
2022-04-26T13:47:05.344 INFO:teuthology.orchestra.console:Power off for smithi119 completed
2022-04-26T13:47:05.446 INFO:teuthology.orchestra.console:Power on smithi119
2022-04-26T13:47:09.565 INFO:teuthology.orchestra.console:Power off for smithi196 completed
2022-04-26T13:47:09.681 INFO:teuthology.orchestra.console:Power on smithi196
2022-04-26T13:47:13.807 INFO:teuthology.orchestra.console:Power on for smithi119 completed
2022-04-26T13:47:13.908 INFO:teuthology.provision.fog.smithi119:Waiting for deploy to finish
2022-04-26T13:47:13.934 INFO:teuthology.orchestra.console:Power on for smithi196 completed
2022-04-26T13:47:14.036 INFO:teuthology.provision.fog.smithi196:Waiting for deploy to finish
2022-04-26T13:49:45.389 ERROR:teuthology.orchestra.connection:Error authenticating with smithi196.front.sepia.ceph.com: Authentication failed.
2022-04-26T13:51:22.522 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-26T13:51:26.143 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-26T13:51:47.788 INFO:teuthology.provision.fog.smithi119:Node is ready
2022-04-26T13:51:47.808 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119.front.sepia.ceph.com
2022-04-26T13:51:47.864 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119 smithi119.front.sepia.ceph.com smithi119
2022-04-26T13:51:48.226 INFO:teuthology.provision.fog.smithi119:Deploy complete!
2022-04-26T13:51:51.771 INFO:teuthology.provision.fog.smithi196:Node is ready
2022-04-26T13:51:51.789 INFO:teuthology.orchestra.run.smithi196.stdout:smithi196.front.sepia.ceph.com
2022-04-26T13:51:51.845 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.15.196 smithi196.front.sepia.ceph.com smithi196
2022-04-26T13:51:52.217 INFO:teuthology.provision.fog.smithi196:Deploy complete!
2022-04-26T13:51:52.377 INFO:teuthology.lock.ops:Checking smithi119.front.sepia.ceph.com
2022-04-26T13:51:52.378 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-26T13:51:52.395 INFO:teuthology.lock.ops:Checking smithi196.front.sepia.ceph.com
2022-04-26T13:51:52.396 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-26T13:51:52.413 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: set os type and version on server
2022-04-26T13:51:52.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com'
2022-04-26T13:51:52.614 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64
2022-04-26T13:51:52.673 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream"
2022-04-26T13:51:52.673 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="8"
2022-04-26T13:51:52.673 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos"
2022-04-26T13:51:52.674 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora"
2022-04-26T13:51:52.674 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="8"
2022-04-26T13:51:52.674 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el8"
2022-04-26T13:51:52.674 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-26T13:51:52.674 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31"
2022-04-26T13:51:52.674 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-26T13:51:52.674 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/"
2022-04-26T13:51:52.674 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-26T13:51:52.674 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-26T13:51:52.675 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-26T13:51:52.676 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server
2022-04-26T13:51:52.694 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: set os type and version on server
2022-04-26T13:51:52.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com'
2022-04-26T13:51:52.890 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64
2022-04-26T13:51:52.947 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream"
2022-04-26T13:51:52.948 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="8"
2022-04-26T13:51:52.948 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos"
2022-04-26T13:51:52.948 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora"
2022-04-26T13:51:52.948 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="8"
2022-04-26T13:51:52.948 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el8"
2022-04-26T13:51:52.948 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-26T13:51:52.948 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31"
2022-04-26T13:51:52.948 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-26T13:51:52.948 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/"
2022-04-26T13:51:52.949 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-26T13:51:52.949 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-26T13:51:52.949 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-26T13:51:52.949 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server
2022-04-26T13:51:53.056 INFO:teuthology.dispatcher.supervisor:Running job 6806244
2022-04-26T13:51:53.057 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-04-26_04:17:03-fs-pacific-distro-default-smithi/6806244 --name teuthology-2022-04-26_04:17:03-fs-pacific-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/teuthology-2022-04-26_04:17:03-fs-pacific-distro-default-smithi/6806244/orig.config.yaml
2022-04-26T13:51:53.060 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-04-26_04:17:03-fs-pacific-distro-default-smithi/6806244
2022-04-26T13:51:53.060 INFO:teuthology.dispatcher.supervisor:Job PID: 23046
2022-04-26T13:51:53.061 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-04-26T20:22:04.659 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-04-26T20:22:04.881 INFO:teuthology.kill:Killing Pids: {23046}
2022-04-26T20:22:07.196 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2022-04-26T20:22:07.196 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2022-04-26T20:22:07.196 INFO:teuthology.misc:Compressing logs...
2022-04-26T20:22:07.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com'
2022-04-26T20:22:07.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com'
2022-04-26T20:22:07.696 INFO:teuthology.misc:Compressing logs...
2022-04-26T20:22:08.062 INFO:teuthology.misc:Compressing logs...
2022-04-26T20:22:08.250 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/cephadm.log: file size changed while zipping
2022-04-26T20:22:19.225 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-04-26T20:22:21.928 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-mds.cephfs.smithi196.qeabgl.log: file size changed while zipping
2022-04-26T20:22:26.980 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-mon.smithi196.log: file size changed while zipping
2022-04-26T20:22:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-mon.smithi119.log: file size changed while zipping
2022-04-26T20:22:37.687 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-04-26T20:22:44.077 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-mds.cephfs.smithi119.gkvexa.log: file size changed while zipping
2022-04-26T20:22:46.960 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-mgr.smithi119.kppqaw.log: file size changed while zipping
2022-04-26T20:22:49.269 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-04-26T20:22:49.641 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-mds.cephfs.smithi119.jchqko.log: file size changed while zipping
2022-04-26T20:22:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-04-26T20:22:58.618 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/0b6017b0-c569-11ec-8c39-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-04-26T20:23:20.122 INFO:teuthology.kill:No teuthology processes running
2022-04-26T20:23:21.404 INFO:teuthology.kill:Nuking machines: ['smithi119', 'smithi196']
2022-04-26T20:23:22.486 INFO:teuthology.kill:2022-04-26 20:23:22,486.486 INFO:teuthology.nuke:targets:
2022-04-26T20:23:22.487 INFO:teuthology.kill:  smithi119.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChTvyeuP8cqtpqttKOvsUEAjDXebJVlPxNiTUoifo/AHFxmw6FSq77rvl6lyx+CflKICBQ58GD0Wxqz+7hi/i+soQKgEtBEYqXVWbLXAlsSyJjR6V9MCmmqyFl3lGsYnHd8IpB8JWKpTAe0xNcJb6lMM/2tFLCnrzRrNpDoncDT/WMU66XGqi5RwmfindxxoBWdhCWBbHLcHCQ85Su1l6z3HsuA6IEFRQCLM4gzqYHWKmcyGyr1eRxa3Foabw64F/JiUj1TrmflHjKCVmrwAU2BqCSJtxf/taT9zQ5B+KhoAN+kO4hv2Y/v3l/dwZb3o4zJKGReK1gdCmzmg3QFhDIl1TG9HQ09BynodkxfAhz0TUkdWVXqQLVPu0xpQ+Tg9oyWJk0Kil6sIkxq8HGDAGCrZmWVR2CkGp2IDCviDgjU4Mfu2p4P00DwPDkJXxgilBdP2zyYb6UJ51vyDQwpkVrNimV9uSGvoSEMcST+XLnbcXY1quq/pNMGyjoS/+nY4M=
2022-04-26T20:23:22.487 INFO:teuthology.kill:  smithi196.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrInZNjDQEGMRVIOJVSJPspicAB1+odzHkRuN88BkuG1NC2TRDq50A9YxKj11zwfbcAmZeOSuccSQQWWQ6w0H7R/uEqO3kj3zco4tI2A+2qCbRVlXaYNK2VA7AhqaoVjTXlMMEvGnvQQau1Iybzs1Y2iVySFr4tTTz7Pi+4Eg3LuE/zq5yN1Sxzo6/4QJVBSoOoHTd40SW15EFqzO9T791hg+VQrRqNUwv9B0Tcrd6AXnH1VuyGmNblBZgSu6CUBaJsOJ1zm8vj9Ep7A5dFwb4YQyLEywQFy7bL6G+m39jrHe9VMnHl0VQ/OoIwsB38wV8RvUk2e0rLFrtzdho2wOMo2Reyyv8HpcjrC+xo69wHx+01r8VsNOYEIS2cnLuNHGIqzpRkovPCBdxo3fiW0/NL+94Y95Gpb+GjAguB50mLAzFYcAKT08zFqoV8zcKpiQYqDD2+p+/JyFpF+t+/NNDgMVhrOqlllbmsZsDxKMooW3QuOQXU2VfIqPdZ4v4Ucc=
2022-04-26T20:23:22.513 INFO:teuthology.kill:2022-04-26 20:23:22,513.513 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-26T20:23:22.516 INFO:teuthology.kill:2022-04-26 20:23:22,516.516 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-26T20:23:22.589 INFO:teuthology.kill:2022-04-26 20:23:22,589.589 INFO:teuthology.orchestra.console:Power off smithi119
2022-04-26T20:23:22.600 INFO:teuthology.kill:2022-04-26 20:23:22,600.600 INFO:teuthology.orchestra.console:Power off smithi196
2022-04-26T20:23:30.961 INFO:teuthology.kill:2022-04-26 20:23:30,961.961 INFO:teuthology.orchestra.console:Power off for smithi196 completed
2022-04-26T20:23:31.122 INFO:teuthology.kill:2022-04-26 20:23:31,121.121 INFO:teuthology.lock.ops:unlocked: smithi196.front.sepia.ceph.com
2022-04-26T20:23:35.067 INFO:teuthology.kill:2022-04-26 20:23:35,066.066 INFO:teuthology.orchestra.console:Power off for smithi119 completed
2022-04-26T20:23:35.213 INFO:teuthology.kill:2022-04-26 20:23:35,212.212 INFO:teuthology.lock.ops:unlocked: smithi119.front.sepia.ceph.com
2022-04-26T20:25:35.400 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15