2022-10-25T21:27:36.192 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-25T21:27:36.332 INFO:teuthology.lock.ops:Start node 'smithi080.front.sepia.ceph.com' reimaging
2022-10-25T21:27:36.334 INFO:teuthology.lock.ops:Updating [smithi080.front.sepia.ceph.com]: reset os type and version on server
2022-10-25T21:27:36.335 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server
2022-10-25T21:27:36.356 INFO:teuthology.lock.ops:Node 'smithi080.front.sepia.ceph.com' reimaging is complete
2022-10-25T21:27:36.357 INFO:teuthology.lock.ops:Start node 'smithi189.front.sepia.ceph.com' reimaging
2022-10-25T21:27:36.357 INFO:teuthology.lock.ops:Updating [smithi189.front.sepia.ceph.com]: reset os type and version on server
2022-10-25T21:27:36.358 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server
2022-10-25T21:27:36.386 INFO:teuthology.lock.ops:Node 'smithi189.front.sepia.ceph.com' reimaging is complete
2022-10-25T21:27:36.576 INFO:teuthology.provision.fog.smithi080:Scheduling deploy of centos 8.stream
2022-10-25T21:27:36.593 INFO:teuthology.provision.fog.smithi189:Scheduling deploy of centos 8.stream
2022-10-25T21:27:36.888 INFO:teuthology.orchestra.console:Power off smithi189
2022-10-25T21:27:36.902 INFO:teuthology.orchestra.console:Power off smithi080
2022-10-25T21:27:45.275 INFO:teuthology.orchestra.console:Power off for smithi080 completed
2022-10-25T21:27:45.377 INFO:teuthology.orchestra.console:Power on smithi080
2022-10-25T21:27:49.397 INFO:teuthology.orchestra.console:Power off for smithi189 completed
2022-10-25T21:27:49.498 INFO:teuthology.orchestra.console:Power on smithi189
2022-10-25T21:27:53.746 INFO:teuthology.orchestra.console:Power on for smithi080 completed
2022-10-25T21:27:53.847 INFO:teuthology.provision.fog.smithi080:Waiting for deploy to finish
2022-10-25T21:27:57.864 INFO:teuthology.orchestra.console:Power on for smithi189 completed
2022-10-25T21:27:57.965 INFO:teuthology.provision.fog.smithi189:Waiting for deploy to finish
2022-10-25T21:30:28.933 ERROR:teuthology.orchestra.connection:Error authenticating with smithi189.front.sepia.ceph.com: Authentication failed.
2022-10-25T21:31:57.316 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-10-25T21:32:22.396 INFO:teuthology.provision.fog.smithi080:Node is ready
2022-10-25T21:32:22.417 INFO:teuthology.orchestra.run.smithi080.stdout:smithi080.front.sepia.ceph.com
2022-10-25T21:32:22.476 INFO:teuthology.orchestra.run.smithi080.stdout:172.21.15.80 smithi080.front.sepia.ceph.com smithi080
2022-10-25T21:32:22.844 INFO:teuthology.provision.fog.smithi080:Deploy complete!
2022-10-25T21:32:42.216 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-10-25T21:32:42.528 INFO:teuthology.provision.fog.smithi189:Node is ready
2022-10-25T21:32:42.547 INFO:teuthology.orchestra.run.smithi189.stdout:smithi189.front.sepia.ceph.com
2022-10-25T21:32:42.605 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189 smithi189.front.sepia.ceph.com smithi189
2022-10-25T21:32:42.953 INFO:teuthology.provision.fog.smithi189:Deploy complete!
2022-10-25T21:32:43.159 INFO:teuthology.lock.ops:Checking smithi080.front.sepia.ceph.com
2022-10-25T21:32:43.160 INFO:teuthology.lock.ops:New key found. Updating...
2022-10-25T21:32:43.183 INFO:teuthology.lock.ops:Checking smithi189.front.sepia.ceph.com
2022-10-25T21:32:43.184 INFO:teuthology.lock.ops:New key found. Updating...
2022-10-25T21:32:43.211 INFO:teuthology.lock.ops:Updating [smithi080.front.sepia.ceph.com]: set os type and version on server
2022-10-25T21:32:43.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com'
2022-10-25T21:32:43.419 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64
2022-10-25T21:32:43.477 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="CentOS Stream"
2022-10-25T21:32:43.478 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="8"
2022-10-25T21:32:43.478 INFO:teuthology.orchestra.run.smithi080.stdout:ID="centos"
2022-10-25T21:32:43.478 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE="rhel fedora"
2022-10-25T21:32:43.479 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="8"
2022-10-25T21:32:43.479 INFO:teuthology.orchestra.run.smithi080.stdout:PLATFORM_ID="platform:el8"
2022-10-25T21:32:43.479 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="CentOS Stream 8"
2022-10-25T21:32:43.479 INFO:teuthology.orchestra.run.smithi080.stdout:ANSI_COLOR="0;31"
2022-10-25T21:32:43.480 INFO:teuthology.orchestra.run.smithi080.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-10-25T21:32:43.480 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://centos.org/"
2022-10-25T21:32:43.480 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-10-25T21:32:43.480 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-10-25T21:32:43.481 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-10-25T21:32:43.482 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server
2022-10-25T21:32:43.505 INFO:teuthology.lock.ops:Updating [smithi189.front.sepia.ceph.com]: set os type and version on server
2022-10-25T21:32:43.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com'
2022-10-25T21:32:43.730 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64
2022-10-25T21:32:43.788 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream"
2022-10-25T21:32:43.789 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="8"
2022-10-25T21:32:43.789 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos"
2022-10-25T21:32:43.790 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora"
2022-10-25T21:32:43.790 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="8"
2022-10-25T21:32:43.791 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el8"
2022-10-25T21:32:43.791 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 8"
2022-10-25T21:32:43.791 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31"
2022-10-25T21:32:43.792 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-10-25T21:32:43.792 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/"
2022-10-25T21:32:43.792 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-10-25T21:32:43.792 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-10-25T21:32:43.793 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-10-25T21:32:43.793 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server
2022-10-25T21:32:43.926 INFO:teuthology.dispatcher.supervisor:Running job 7080235
2022-10-25T21:32:43.926 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-10-25_03:15:06-fs-quincy-distro-default-smithi/7080235 --name teuthology-2022-10-25_03:15:06-fs-quincy-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/teuthology-2022-10-25_03:15:06-fs-quincy-distro-default-smithi/7080235/orig.config.yaml
2022-10-25T21:32:43.931 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-10-25_03:15:06-fs-quincy-distro-default-smithi/7080235
2022-10-25T21:32:43.932 INFO:teuthology.dispatcher.supervisor:Job PID: 29081
2022-10-25T21:32:43.933 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-10-26T09:33:04.809 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-10-26T09:33:06.228 INFO:teuthology.kill:Killing Pids: {29081}
2022-10-26T09:33:07.576 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2022-10-26T09:33:07.577 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2022-10-26T09:33:07.577 INFO:teuthology.misc:Compressing logs...
2022-10-26T09:33:07.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com'
2022-10-26T09:33:07.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com'
2022-10-26T09:33:08.110 INFO:teuthology.misc:Compressing logs...
2022-10-26T09:33:08.421 INFO:teuthology.misc:Compressing logs...
2022-10-26T09:33:12.440 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-mon.smithi189.log: file size changed while zipping
2022-10-26T09:33:15.220 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-10-26T09:33:15.752 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-mds.cephfs.smithi080.mqpgyh.log: file size changed while zipping
2022-10-26T09:33:20.069 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-10-26T09:33:22.014 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-10-26T09:33:26.144 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-10-26T09:33:27.655 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-10-26T09:33:32.079 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-mgr.smithi080.szmewu.log: file size changed while zipping
2022-10-26T09:33:32.523 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-10-26T09:33:33.351 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-mds.cephfs.smithi189.bwfhev.log: file size changed while zipping
2022-10-26T09:33:42.798 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/ebb71128-54ad-11ed-8438-001a4aab830c/ceph-mon.smithi080.log: file size changed while zipping
2022-10-26T09:34:30.681 INFO:teuthology.kill:No teuthology processes running
2022-10-26T09:34:31.868 INFO:teuthology.kill:Nuking machines: ['smithi080', 'smithi189']
2022-10-26T09:34:32.863 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-10-26T09:34:32.864 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-10-26T09:34:32.864 INFO:teuthology.kill:2022-10-26 09:34:32,863.863 INFO:teuthology.nuke:targets:
2022-10-26T09:34:32.865 INFO:teuthology.kill:  smithi080.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7xENijAK/YNsoWbwaOnxc9RYRxlR9ghAfKEkA9101gXsC6Z8CCUkf11jUjGNXxFw/KKcdGhggQVu8rCGRm5FfJfULoClBwGqeIOiMK8qb5CoCCoX1Vo0pd1t4I/AXtqw8kFjXRrg58L7t5wEPQtnjW67tq7kNrexXjrQtl0BRfJG8Vsanql3h1LW/6bK/vDUUXjMlRxeoFlryNAIb73bXvCRGe2JN9STBmv7cWwQxkHnFmoSiFeSFkzp66GtDRw8RUMdFbrqMfCEnmxhyU2xTgom8jMrbaH1uKaf1qZ1n1mCgJdhDBjnsGsUBpYw1krGe2qH+BK7BVrY9aSjBCOVUFwdPhOUEGl37ok302ARMHcWPD7Edtrhqj/sb35L1cgXGjm1yTvwi0322PiO/fR5muj41Q+n4E3caEi1RlSKJeTKYEBGaPSCUd18TbmXEP50C5kAfbj8OglNxemKdxAKSSetHNb9M7nSZFSVF5arE3dW5oCK9RnOAjjCT2U1onxc=
2022-10-26T09:34:32.865 INFO:teuthology.kill:  smithi189.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2Qv21zr03fWaUdImYAkcJREGWIriZQuV3EiyKAiEtfR3SbRIt1gRhXan8SQ3QsQbUUkJ0S/xj2NqcnUJGjc79bIjkrG/VeKgy9QCNWvfrCWy+uACCRbCcpbdfe+lp2XttEgaZzaVesvqYRQcWRrknEhnrev6COdYj6jjizDIrNnD9CgldrJgtSUYrt9YqcRiVb5On2VnC/g3bGV8K3f69ufr7tdfaOigKwlil7gMz08GUrNxAa1msav7FCx0Pkpa292LLBLf7hz++hHdn/zh5EGu+lPTRh9oyVWJUnLWwPUh/NYMCiaWJGrHAafdYLnXrxBjFKWllZ5AKz4GMFI1nYqc2+XsiXkijkHakO+VAeIxJpdGIuK06OTFuRHI8Nh6NnbY4RpQWWQsjIfwN97hMRa4LtgLBFzGH1EuIqA1Y4Bt2wFTUa2KNahnhuesbDopmWPD2MtuZquhr2bJ+YgpBW9QW5dsySq1oBum9irMzlGV+biI1m7OwqdEhb/7q1Ck=
2022-10-26T09:34:32.865 INFO:teuthology.kill:2022-10-26 09:34:32,863.863 INFO:teuthology.nuke:Checking targets against current locks
2022-10-26T09:34:32.910 INFO:teuthology.kill:2022-10-26 09:34:32,910.910 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-10-26T09:34:32.936 INFO:teuthology.kill:2022-10-26 09:34:32,936.936 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-10-26T09:34:32.991 INFO:teuthology.kill:2022-10-26 09:34:32,990.990 INFO:teuthology.orchestra.console:Power off smithi080
2022-10-26T09:34:33.025 INFO:teuthology.kill:2022-10-26 09:34:33,024.024 INFO:teuthology.orchestra.console:Power off smithi189
2022-10-26T09:34:41.359 INFO:teuthology.kill:2022-10-26 09:34:41,359.359 INFO:teuthology.orchestra.console:Power off for smithi080 completed
2022-10-26T09:34:41.508 INFO:teuthology.kill:2022-10-26 09:34:41,507.507 INFO:teuthology.lock.ops:unlocked: smithi080.front.sepia.ceph.com
2022-10-26T09:34:45.489 INFO:teuthology.kill:2022-10-26 09:34:45,488.488 INFO:teuthology.orchestra.console:Power off for smithi189 completed
2022-10-26T09:34:45.635 INFO:teuthology.kill:2022-10-26 09:34:45,634.634 INFO:teuthology.lock.ops:unlocked: smithi189.front.sepia.ceph.com
2022-10-26T09:36:45.837 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-10-26T09:36:45.951 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi080 but it was locked by another job: /home/teuthworker/archive/dis-2022-10-25_23:10:37-rbd-wip-dis-testing-distro-default-smithi/7080894
2022-10-26T09:36:45.952 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi189 but it was locked by another job: /home/teuthworker/archive/dis-2022-10-25_23:10:37-rbd-wip-dis-testing-distro-default-smithi/7080894