2022-07-28T07:51:09.422 INFO:root:teuthology version: 1.1.1.dev399+g5f965ce1 2022-07-28T07:51:09.571 INFO:teuthology.lock.ops:Start node 'smithi170.front.sepia.ceph.com' reimaging 2022-07-28T07:51:09.573 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: reset os type and version on server 2022-07-28T07:51:09.574 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2022-07-28T07:51:09.597 INFO:teuthology.lock.ops:Node 'smithi170.front.sepia.ceph.com' reimaging is complete 2022-07-28T07:51:09.597 INFO:teuthology.lock.ops:Start node 'smithi172.front.sepia.ceph.com' reimaging 2022-07-28T07:51:09.598 INFO:teuthology.lock.ops:Updating [smithi172.front.sepia.ceph.com]: reset os type and version on server 2022-07-28T07:51:09.598 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2022-07-28T07:51:09.619 INFO:teuthology.lock.ops:Node 'smithi172.front.sepia.ceph.com' reimaging is complete 2022-07-28T07:51:09.826 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of centos 8.stream 2022-07-28T07:51:09.840 INFO:teuthology.provision.fog.smithi172:Scheduling deploy of centos 8.stream 2022-07-28T07:51:10.250 INFO:teuthology.orchestra.console:Power off smithi172 2022-07-28T07:51:10.265 INFO:teuthology.orchestra.console:Power off smithi170 2022-07-28T07:51:10.398 INFO:teuthology.orchestra.console:Power off for smithi172 completed 2022-07-28T07:51:10.420 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2022-07-28T07:51:10.499 INFO:teuthology.orchestra.console:Power on smithi172 2022-07-28T07:51:10.521 INFO:teuthology.orchestra.console:Power on smithi170 2022-07-28T07:51:18.880 INFO:teuthology.orchestra.console:Power on for smithi172 completed 2022-07-28T07:51:18.886 INFO:teuthology.orchestra.console:Power on for smithi170 completed 2022-07-28T07:51:18.982 INFO:teuthology.provision.fog.smithi172:Waiting for deploy to finish 2022-07-28T07:51:18.986 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish 2022-07-28T07:55:28.648 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-28T07:55:46.867 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-28T07:55:53.823 INFO:teuthology.provision.fog.smithi170:Node is ready 2022-07-28T07:55:53.844 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com 2022-07-28T07:55:53.902 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170 2022-07-28T07:55:54.255 INFO:teuthology.provision.fog.smithi170:Deploy complete! 2022-07-28T07:56:12.454 INFO:teuthology.provision.fog.smithi172:Node is ready 2022-07-28T07:56:12.479 INFO:teuthology.orchestra.run.smithi172.stdout:smithi172.front.sepia.ceph.com 2022-07-28T07:56:12.540 INFO:teuthology.orchestra.run.smithi172.stdout:172.21.15.172 smithi172.front.sepia.ceph.com smithi172 2022-07-28T07:56:12.914 INFO:teuthology.provision.fog.smithi172:Deploy complete! 2022-07-28T07:56:13.097 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com 2022-07-28T07:56:13.098 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-28T07:56:13.122 INFO:teuthology.lock.ops:Checking smithi172.front.sepia.ceph.com 2022-07-28T07:56:13.123 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-28T07:56:13.143 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server 2022-07-28T07:56:13.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2022-07-28T07:56:13.414 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2022-07-28T07:56:13.472 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2022-07-28T07:56:13.473 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8" 2022-07-28T07:56:13.473 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2022-07-28T07:56:13.474 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2022-07-28T07:56:13.474 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8" 2022-07-28T07:56:13.475 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8" 2022-07-28T07:56:13.475 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-28T07:56:13.475 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2022-07-28T07:56:13.475 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-28T07:56:13.476 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2022-07-28T07:56:13.476 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-28T07:56:13.476 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-28T07:56:13.476 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-28T07:56:13.478 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2022-07-28T07:56:13.502 INFO:teuthology.lock.ops:Updating [smithi172.front.sepia.ceph.com]: set os type and version on server 2022-07-28T07:56:13.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2022-07-28T07:56:13.720 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2022-07-28T07:56:13.778 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2022-07-28T07:56:13.779 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8" 2022-07-28T07:56:13.779 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2022-07-28T07:56:13.779 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2022-07-28T07:56:13.780 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8" 2022-07-28T07:56:13.780 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2022-07-28T07:56:13.780 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-28T07:56:13.780 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2022-07-28T07:56:13.781 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-28T07:56:13.781 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2022-07-28T07:56:13.781 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-28T07:56:13.781 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-28T07:56:13.782 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-28T07:56:13.782 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2022-07-28T07:56:13.912 INFO:teuthology.dispatcher.supervisor:Running job 6951646 2022-07-28T07:56:13.913 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_5f965ce157b8dcdaf2da92a5b5d1a63e787efdd9/virtualenv/bin/teuthology -v --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2022-07-28_05:39:24-orch:cephadm-wip-adk3-testing-2022-07-27-2041-quincy-distro-default-smithi/6951646 --name gabrioux-2022-07-28_05:39:24-orch:cephadm-wip-adk3-testing-2022-07-27-2041-quincy-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/gabrioux-2022-07-28_05:39:24-orch:cephadm-wip-adk3-testing-2022-07-27-2041-quincy-distro-default-smithi/6951646/orig.config.yaml 2022-07-28T07:56:13.917 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/gabrioux-2022-07-28_05:39:24-orch:cephadm-wip-adk3-testing-2022-07-27-2041-quincy-distro-default-smithi/6951646 2022-07-28T07:56:13.918 INFO:teuthology.dispatcher.supervisor:Job PID: 2922 2022-07-28T07:56:13.919 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-07-28T19:57:13.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-07-28T19:57:14.248 INFO:teuthology.kill:Killing Pids: {2922} 2022-07-28T19:57:16.024 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-07-28T19:57:16.025 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-07-28T19:57:16.025 INFO:teuthology.misc:Compressing logs... 2022-07-28T19:57:16.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2022-07-28T19:57:16.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2022-07-28T19:57:16.535 INFO:teuthology.misc:Compressing logs... 2022-07-28T19:57:16.862 INFO:teuthology.misc:Compressing logs... 2022-07-28T19:57:23.915 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-mgr.smithi170.jislzo.log: file size changed while zipping 2022-07-28T19:57:24.384 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-mon.smithi172.log: file size changed while zipping 2022-07-28T19:57:41.390 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-07-28T19:57:55.934 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-07-28T19:58:12.429 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-07-28T19:58:31.524 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-mon.smithi170.log: file size changed while zipping 2022-07-28T19:59:03.567 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-mds.cephfs.smithi172.aapabt.log: file size changed while zipping 2022-07-28T19:59:20.526 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-07-28T19:59:40.833 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-07-28T19:59:55.959 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/17e3c8c0-0e4c-11ed-8430-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-07-28T20:00:44.440 INFO:teuthology.kill:No teuthology processes running 2022-07-28T20:00:46.004 INFO:teuthology.kill:Nuking machines: ['smithi170', 'smithi172'] 2022-07-28T20:00:47.357 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_5f965ce157b8dcdaf2da92a5b5d1a63e787efdd9/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-07-28T20:00:47.358 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-07-28T20:00:47.358 INFO:teuthology.kill:2022-07-28 20:00:47,357.357 INFO:teuthology.nuke:targets: 2022-07-28T20:00:47.359 INFO:teuthology.kill: smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDeF5LL14MATyhQsRUM1NOh+YEZMgduULUAli0q4/ng+nF4CMQe5sNYnHbaXZ9EBaZCWcaCD36gyPM63pxLm92dmyn99jP+0bISbQTTHCa1lAZp6ldNn/9PexVn7Gpa/Di4GWUMeMB34Or190EHNMPZLqN5IiMQMrtoHb0f/HFPH9ECNt9IhCr+UBuOoCvndsQFdmWe+bZMtFEwLeYH5x1wBjm8USxuLt7m6Wd84EaaPtBBnV4zAV5nF85oXlPUWX4T9E9n3vC/d/vQcjoPV1nT4Oe3gpfFkG4dfhGepr054EELuX+5dPdLVFL/IabJ6jABYO96pp62Ndcy78aP2rZ0F3V4cI7BQWcVSDmv8iksceKT8NC1rzFfjuOrLyqH4XcJNlvH/GOAJEkVKXd4/W1uI/zwSo6xWY+/G/1/u8BvUbTmTaHfJG4gPIOYnbCMipiqo8G7yfpWXeWW22+LROFf/NZOv4daXUgY9RYpsN+Uuw7Ev34sYzPcGtwuA7sFECU= 2022-07-28T20:00:47.359 INFO:teuthology.kill: smithi172.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDa4S9Nbg08SxtPThjI1eG5HiteU2xLk4txz2yUTjIOIkHX6dIZ4AmhdJQJo3rMVgw2PBBz5wkKX1mvs439TJ7QT3uNKaWTUneMRxZukWULbIxbD8MVn6xz2zKK+9f9dkDOa+WCnsiA5sVCgjVz6YXIZZ+amTkE0gVQEKBs4zYeaHbCXV21GA+LyjhePaU+zzvNtwlZZfSyQt5z7Vw0fuMc7QAaTLZwdE5+P1gWWdqBH3GEI2hxkHq1UCSSg2lozofB8Xau975t4jJx3RQl6HdClGYDA9/G3yswyBogdB8FT7pjA0ZJG4hNyaQiT0qZdtLlNJi8knPqQmaJ8kQdUKKcDVDCeEEfafOiGOABqot0QZr+41i8j4dHnM4FCQf7lmnICOiM0ttx8Nb3d24IyNnydyI7p+ovZkmRx4IYgAX3yEOuPWDwkfi8LENSwUq77jKsShbSo2+SJRHrXiqv+M0XPhAyy53DNfqAoYNrb+V+0b7cNHaoHOmE8R1kKOPoBE= 2022-07-28T20:00:47.359 INFO:teuthology.kill:2022-07-28 20:00:47,358.358 INFO:teuthology.nuke:Checking targets against current locks 2022-07-28T20:00:47.445 INFO:teuthology.kill:2022-07-28 20:00:47,444.444 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-28T20:00:47.478 INFO:teuthology.kill:2022-07-28 20:00:47,477.477 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-28T20:00:47.564 INFO:teuthology.kill:2022-07-28 20:00:47,564.564 INFO:teuthology.orchestra.console:Power off smithi170 2022-07-28T20:00:47.643 INFO:teuthology.kill:2022-07-28 20:00:47,642.642 INFO:teuthology.orchestra.console:Power off smithi172 2022-07-28T20:00:56.035 INFO:teuthology.kill:2022-07-28 20:00:56,034.034 INFO:teuthology.orchestra.console:Power off for smithi172 completed 2022-07-28T20:00:56.192 INFO:teuthology.kill:2022-07-28 20:00:56,191.191 INFO:teuthology.lock.ops:unlocked: smithi172.front.sepia.ceph.com 2022-07-28T20:01:00.100 INFO:teuthology.kill:2022-07-28 20:01:00,099.099 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2022-07-28T20:01:00.259 INFO:teuthology.kill:2022-07-28 20:01:00,259.259 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com 2022-07-28T20:03:00.675 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-07-28T20:03:00.804 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi172 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-07-28_15:17:14-fs-wip-yuri-testing-2022-07-25-0645-quincy-distro-default-smithi/6952520