2022-07-06T13:21:28.664 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-06T13:21:28.875 INFO:teuthology.lock.ops:Start node 'smithi073.front.sepia.ceph.com' reimaging 2022-07-06T13:21:28.877 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: reset os type and version on server 2022-07-06T13:21:28.878 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2022-07-06T13:21:28.903 INFO:teuthology.lock.ops:Node 'smithi073.front.sepia.ceph.com' reimaging is complete 2022-07-06T13:21:28.903 INFO:teuthology.lock.ops:Start node 'smithi148.front.sepia.ceph.com' reimaging 2022-07-06T13:21:28.904 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: reset os type and version on server 2022-07-06T13:21:28.905 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2022-07-06T13:21:28.938 INFO:teuthology.lock.ops:Node 'smithi148.front.sepia.ceph.com' reimaging is complete 2022-07-06T13:21:29.171 INFO:teuthology.provision.fog.smithi073:Scheduling deploy of centos 8.stream 2022-07-06T13:21:29.180 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of centos 8.stream 2022-07-06T13:21:29.590 INFO:teuthology.orchestra.console:Power off smithi148 2022-07-06T13:21:29.611 INFO:teuthology.orchestra.console:Power off smithi073 2022-07-06T13:21:37.998 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2022-07-06T13:21:38.100 INFO:teuthology.orchestra.console:Power on smithi073 2022-07-06T13:21:42.127 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2022-07-06T13:21:42.231 INFO:teuthology.orchestra.console:Power on smithi148 2022-07-06T13:21:46.485 INFO:teuthology.orchestra.console:Power on for smithi073 completed 2022-07-06T13:21:46.587 INFO:teuthology.provision.fog.smithi073:Waiting for deploy to finish 2022-07-06T13:21:50.609 INFO:teuthology.orchestra.console:Power on for smithi148 completed 2022-07-06T13:21:50.712 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish 2022-07-06T13:26:27.615 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-06T13:26:27.920 INFO:teuthology.provision.fog.smithi073:Node is ready 2022-07-06T13:26:27.940 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073.front.sepia.ceph.com 2022-07-06T13:26:28.005 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73 smithi073.front.sepia.ceph.com smithi073 2022-07-06T13:26:28.359 INFO:teuthology.provision.fog.smithi073:Deploy complete! 2022-07-06T13:26:55.740 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-06T13:26:56.090 INFO:teuthology.provision.fog.smithi148:Node is ready 2022-07-06T13:26:56.144 INFO:teuthology.orchestra.run.smithi148.stdout:smithi148.front.sepia.ceph.com 2022-07-06T13:26:56.220 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148 smithi148.front.sepia.ceph.com smithi148 2022-07-06T13:26:56.658 INFO:teuthology.provision.fog.smithi148:Deploy complete! 2022-07-06T13:26:56.873 INFO:teuthology.lock.ops:Checking smithi073.front.sepia.ceph.com 2022-07-06T13:26:56.874 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-06T13:26:56.904 INFO:teuthology.lock.ops:Checking smithi148.front.sepia.ceph.com 2022-07-06T13:26:56.905 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-06T13:26:56.934 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: set os type and version on server 2022-07-06T13:26:56.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2022-07-06T13:26:57.271 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2022-07-06T13:26:57.339 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream" 2022-07-06T13:26:57.339 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="8" 2022-07-06T13:26:57.339 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2022-07-06T13:26:57.340 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2022-07-06T13:26:57.340 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="8" 2022-07-06T13:26:57.340 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el8" 2022-07-06T13:26:57.340 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-06T13:26:57.341 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2022-07-06T13:26:57.341 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-06T13:26:57.341 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/" 2022-07-06T13:26:57.341 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-06T13:26:57.342 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-06T13:26:57.342 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-06T13:26:57.344 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2022-07-06T13:26:57.366 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: set os type and version on server 2022-07-06T13:26:57.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2022-07-06T13:26:57.595 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2022-07-06T13:26:57.668 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2022-07-06T13:26:57.669 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8" 2022-07-06T13:26:57.669 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2022-07-06T13:26:57.669 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2022-07-06T13:26:57.670 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2022-07-06T13:26:57.670 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2022-07-06T13:26:57.671 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-06T13:26:57.671 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2022-07-06T13:26:57.672 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-06T13:26:57.672 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2022-07-06T13:26:57.673 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-06T13:26:57.673 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-06T13:26:57.673 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-06T13:26:57.675 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2022-07-06T13:26:57.849 INFO:teuthology.dispatcher.supervisor:Running job 6917375 2022-07-06T13:26:57.849 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/bin/teuthology -v --owner scheduled_dparmar@teuthology --archive /home/teuthworker/archive/dparmar-2022-07-06_12:57:28-fs:upgrade-wip-dparmar-cephadm-55715-distro-default-smithi/6917375 --name dparmar-2022-07-06_12:57:28-fs:upgrade-wip-dparmar-cephadm-55715-distro-default-smithi --description fs:upgrade/upgrade_without_reducing_max_mds/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/dparmar-2022-07-06_12:57:28-fs:upgrade-wip-dparmar-cephadm-55715-distro-default-smithi/6917375/orig.config.yaml 2022-07-06T13:26:57.857 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/dparmar-2022-07-06_12:57:28-fs:upgrade-wip-dparmar-cephadm-55715-distro-default-smithi/6917375 2022-07-06T13:26:57.858 INFO:teuthology.dispatcher.supervisor:Job PID: 20057 2022-07-06T13:26:57.858 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-07-07T01:28:06.112 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-07-07T01:28:06.498 INFO:teuthology.kill:Killing Pids: {20057} 2022-07-07T01:28:09.238 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-07-07T01:28:09.240 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-07-07T01:28:09.240 INFO:teuthology.misc:Compressing logs... 2022-07-07T01:28:09.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2022-07-07T01:28:09.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2022-07-07T01:28:09.824 INFO:teuthology.misc:Compressing logs... 2022-07-07T01:28:10.001 INFO:teuthology.orchestra.run.smithi148.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-07-07T01:28:10.195 INFO:teuthology.misc:Compressing logs... 2022-07-07T01:28:15.303 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-mgr.smithi073.wclvxt.log: file size changed while zipping 2022-07-07T01:28:18.244 INFO:teuthology.orchestra.run.smithi148.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-mon.smithi148.log: file size changed while zipping 2022-07-07T01:28:34.842 INFO:teuthology.orchestra.run.smithi148.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-07-07T01:28:37.501 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-mon.smithi073.log: file size changed while zipping 2022-07-07T01:28:50.500 INFO:teuthology.orchestra.run.smithi148.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-07-07T01:28:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-07-07T01:29:09.328 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-07-07T01:29:27.122 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-07-07T01:29:36.397 INFO:teuthology.orchestra.run.smithi148.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-mds.cephfs.smithi148.iuzyum.log: file size changed while zipping 2022-07-07T01:29:56.898 INFO:teuthology.orchestra.run.smithi148.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-07-07T01:30:25.732 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/da2ee284-fd30-11ec-842d-001a4aab830c/ceph-mds.cephfs.smithi073.pixspp.log: file size changed while zipping 2022-07-07T01:31:20.811 INFO:teuthology.kill:No teuthology processes running 2022-07-07T01:31:22.489 INFO:teuthology.kill:Nuking machines: ['smithi073', 'smithi148'] 2022-07-07T01:31:24.017 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-07-07T01:31:24.018 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-07-07T01:31:24.018 INFO:teuthology.kill:2022-07-07 01:31:24,016.016 INFO:teuthology.nuke:targets: 2022-07-07T01:31:24.019 INFO:teuthology.kill: smithi073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjgOcqz6ii6YFEa+UuhUhAs1eERhHX/rointpu69NyzemvIBgYXHjQl/t5pLjC8yI/0lgxXZtp5wW4rpSCQq1QAznEW0xjIXyvXzQQBJflVQ1JvkW/stOYb6g5uFGReazNMz9QPHvkKLX/nCs82X2M1sxm7pts79hAwCOX/ahcXQUaj6X8kcaUTfN5Cpn7mRWMcmXAtSLa9X9+ozGjvJ7JASDtntsaYG6/9ADHPqk2novVAq8EWBkV6yvF4CweLfPkJMV4K9w9qct1WSXeFeEXu2lE91fo+vXtluzl7UzOq9RhPKg/XEN9O0xsv1FP4nnKsIsl5qf0fX849lxmIT0QqYbxvuYjp3lxL3USbT411XYerKi04f9xJ0/Kmc0zzS3P0dyZSl3mcETzPfSBsFaLoYhF/QRuRSSIev1xPZbQ4ChSAPQMvV538aP3wNDNhZWRRLdR/ySYXvQ9kKTngExZmEU/b6n5zPQKwaQCySJdbaBoLkF+gcqWvP49x+847y0= 2022-07-07T01:31:24.020 INFO:teuthology.kill: smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMP8xU1CV/RR55bzg4QlUTQH7Zt27nnkn44yHlwpzY3y1Ncv8QgJq16Kq9kgCcN4F2lJVEIfARA8YLC5fse9l/y/XATqdcBQdfWW4EZasfh1n/WFzVpuEP6RDjNBWDVKN75R4ShjY2Qwsba9/4U/EV/SJjf/VjTFESaq2E+Iq1rgLXPuZR6c4vIjgV+7TANLuGni+MZ0d4US80OZqV4gz/U8ajNcd1FKevmCyfl6Joe4qu8CdyXqGsg3QqICoVUZBuNagBsz+EBE99i9J5OgQhJndhaM7cHlQUqAH/celmSswC3kGlaQ3pGj+M9pCVFQkDqUbYKnuidlwMg/FN7mQdRvK8DBfUoHTXsT906veGx+BeL5DkbQWqKBOo5u/OzXGwH6CXcu+072tb9khRG08xJxTopQupsYgwf/LN9xebjnbQINnmC195ibv+zFrfHaXgV/bsbLIwjMtp6jBSe8uCgQsZyRd9KfkgERkR/5sYUCTPe5d4vitTo9P3IJoyY2s= 2022-07-07T01:31:24.020 INFO:teuthology.kill:2022-07-07 01:31:24,016.016 INFO:teuthology.nuke:Checking targets against current locks 2022-07-07T01:31:24.120 INFO:teuthology.kill:2022-07-07 01:31:24,120.120 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-07T01:31:24.171 INFO:teuthology.kill:2022-07-07 01:31:24,171.171 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-07T01:31:24.275 INFO:teuthology.kill:2022-07-07 01:31:24,275.275 INFO:teuthology.orchestra.console:Power off smithi073 2022-07-07T01:31:24.339 INFO:teuthology.kill:2022-07-07 01:31:24,338.338 INFO:teuthology.orchestra.console:Power off smithi148 2022-07-07T01:31:32.758 INFO:teuthology.kill:2022-07-07 01:31:32,758.758 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2022-07-07T01:31:32.924 INFO:teuthology.kill:2022-07-07 01:31:32,923.923 INFO:teuthology.lock.ops:unlocked: smithi073.front.sepia.ceph.com 2022-07-07T01:31:36.881 INFO:teuthology.kill:2022-07-07 01:31:36,881.881 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2022-07-07T01:31:37.053 INFO:teuthology.kill:2022-07-07 01:31:37,053.053 INFO:teuthology.lock.ops:unlocked: smithi148.front.sepia.ceph.com 2022-07-07T01:33:37.590 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-07-07T01:33:37.747 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi073 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-07-07_00:16:01-fs-octopus-release-distro-default-smithi/6918038 2022-07-07T01:33:37.747 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi148 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-07-07_00:16:01-fs-octopus-release-distro-default-smithi/6918038