2021-11-08T19:38:35.899 INFO:root:teuthology version: 1.1.0-27954452 2021-11-08T19:38:36.019 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging 2021-11-08T19:38:36.020 INFO:teuthology.lock.ops:Updating [ubuntu@smithi072.front.sepia.ceph.com]: reset os type and version on server 2021-11-08T19:38:36.020 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2021-11-08T19:38:36.040 INFO:teuthology.lock.ops:Node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging is complete 2021-11-08T19:38:36.040 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging 2021-11-08T19:38:36.041 INFO:teuthology.lock.ops:Updating [ubuntu@smithi142.front.sepia.ceph.com]: reset os type and version on server 2021-11-08T19:38:36.041 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-11-08T19:38:36.059 INFO:teuthology.lock.ops:Node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging is complete 2021-11-08T19:38:36.276 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of centos 8.stream 2021-11-08T19:38:36.280 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of centos 8.stream 2021-11-08T19:38:36.667 INFO:teuthology.orchestra.console:Power off smithi142 2021-11-08T19:38:36.682 INFO:teuthology.orchestra.console:Power off smithi072 2021-11-08T19:38:45.037 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-11-08T19:38:45.138 INFO:teuthology.orchestra.console:Power on smithi142 2021-11-08T19:38:49.157 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2021-11-08T19:38:49.258 INFO:teuthology.orchestra.console:Power on smithi072 2021-11-08T19:38:53.623 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2021-11-08T19:38:53.724 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2021-11-08T19:38:57.621 INFO:teuthology.orchestra.console:Power on for smithi072 completed 2021-11-08T19:38:57.722 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish 2021-11-08T19:43:06.200 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-08T19:43:21.584 INFO:teuthology.provision.fog.smithi142:Node is ready 2021-11-08T19:43:21.603 INFO:teuthology.orchestra.run.smithi142.stdout:smithi047 2021-11-08T19:43:21.816 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2021-11-08T19:44:14.912 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-08T19:44:15.289 INFO:teuthology.provision.fog.smithi072:Node is ready 2021-11-08T19:44:15.305 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072.front.sepia.ceph.com 2021-11-08T19:44:15.363 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072 2021-11-08T19:44:15.709 INFO:teuthology.provision.fog.smithi072:Deploy complete! 2021-11-08T19:44:15.862 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com 2021-11-08T19:44:15.863 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-08T19:44:15.884 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com 2021-11-08T19:44:15.885 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-08T19:44:15.906 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server 2021-11-08T19:44:15.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2021-11-08T19:44:16.129 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2021-11-08T19:44:16.184 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2021-11-08T19:44:16.185 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8" 2021-11-08T19:44:16.185 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2021-11-08T19:44:16.185 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2021-11-08T19:44:16.185 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8" 2021-11-08T19:44:16.186 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8" 2021-11-08T19:44:16.186 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-08T19:44:16.187 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2021-11-08T19:44:16.187 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-08T19:44:16.187 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2021-11-08T19:44:16.188 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-08T19:44:16.188 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-08T19:44:16.189 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-08T19:44:16.191 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2021-11-08T19:44:16.209 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2021-11-08T19:44:16.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2021-11-08T19:44:16.448 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2021-11-08T19:44:16.504 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2021-11-08T19:44:16.504 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8" 2021-11-08T19:44:16.504 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2021-11-08T19:44:16.505 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2021-11-08T19:44:16.505 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8" 2021-11-08T19:44:16.505 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8" 2021-11-08T19:44:16.505 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-08T19:44:16.506 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2021-11-08T19:44:16.506 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-08T19:44:16.506 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2021-11-08T19:44:16.506 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-08T19:44:16.507 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-08T19:44:16.507 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-08T19:44:16.507 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-11-08T19:44:16.614 INFO:teuthology.dispatcher.supervisor:Running job 6491476 2021-11-08T19:44:16.614 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-11-08_19:29:02-orch:cephadm-master-distro-basic-smithi/6491476 --name sage-2021-11-08_19:29:02-orch:cephadm-master-distro-basic-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/sage-2021-11-08_19:29:02-orch:cephadm-master-distro-basic-smithi/6491476/orig.config.yaml 2021-11-08T19:44:16.618 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-11-08_19:29:02-orch:cephadm-master-distro-basic-smithi/6491476 2021-11-08T19:44:16.619 INFO:teuthology.dispatcher.supervisor:Job PID: 2763 2021-11-08T19:44:16.619 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-09T07:45:10.294 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-09T07:45:10.526 INFO:teuthology.kill:Killing Pids: {2763} 2021-11-09T07:45:11.841 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-11-09T07:45:11.842 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-11-09T07:45:11.842 INFO:teuthology.misc:Compressing logs... 2021-11-09T07:45:11.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2021-11-09T07:45:12.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2021-11-09T07:45:12.459 INFO:teuthology.misc:Compressing logs... 2021-11-09T07:45:12.605 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-11-09T07:45:12.757 INFO:teuthology.misc:Compressing logs... 2021-11-09T07:45:15.445 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/31d78830-40ce-11ec-8c2c-001a4aab830c/ceph-mgr.smithi072.bvqwdd.log: file size changed while zipping 2021-11-09T07:45:17.827 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/31d78830-40ce-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-11-09T07:45:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/31d78830-40ce-11ec-8c2c-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-11-09T07:45:22.011 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/31d78830-40ce-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-11-09T07:45:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/31d78830-40ce-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-11-09T07:45:24.416 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/31d78830-40ce-11ec-8c2c-001a4aab830c/ceph-mon.smithi142.log: file size changed while zipping 2021-11-09T07:45:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/31d78830-40ce-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-11-09T07:45:29.593 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/31d78830-40ce-11ec-8c2c-001a4aab830c/ceph-mon.smithi072.log: file size changed while zipping 2021-11-09T07:45:34.044 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/31d78830-40ce-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-11-09T07:45:55.113 INFO:teuthology.kill:No teuthology processes running 2021-11-09T07:45:56.384 INFO:teuthology.kill:Nuking machines: ['smithi072', 'smithi142'] 2021-11-09T07:45:57.396 INFO:teuthology.kill:2021-11-09 07:45:57,396.396 INFO:teuthology.nuke:targets: 2021-11-09T07:45:57.397 INFO:teuthology.kill: smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeS9DD+SpuTvP5DEA5xuyUtJXEAkNgvo2DBOxMYQFkJMhglzi3EHAsT5srghvmQBQq4vcT8P0T2VxgwUDylIoLkNDyXpfS52ANnbMQwt/ffgI+a7M71Sea9aO48r4xF8jgNUV+9QDEo1XTmlS61iPm5Eb+5eY5cctmogmITlEX0wJ/FuYUzABHYuQjWEM82iDEiTTtTY1lr5bHmpNmxkbo+w85fkjzHQYfr1h6CNscHL2W82N/WDSvLC3TgHRgrmTtQsfxwcQQTYzH1gnGFn3tH47Jlbu3BdFVHhilXwih05UyZHISy9J9XL5wynZPgpE9Ayk1waWUCDX2U6cr8YADj66LFPniidEM9VNwScMSzpCn688omBObNrui6L7lU8KZxWb73eY371JQWIo+g/SqRrqGxjkgkO9a2bT6LGcOPOMWEydS3aqDouwyvX/ijYfKOYploWvQItl3JMJDurloeGR15NaorehGAaCIMlnBEl5isI/Pu7noN0OqeZCrMy0= 2021-11-09T07:45:57.397 INFO:teuthology.kill: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHG66QaicHnFZXwkTAx0ceAwuNmiBDMkOio8AMWk75bqoyYw9sSqPCo2eijIDXwEnavoslHXq9jptGb6uECsIub7NM5KYnBruGXL0SpGzV+qQ9yfhFnEwGrOoRzlIYq8UbROOSWE5u0GF/CcAJxw7C2UNLGw5NGIbusFe6AVN5r7dhnI6ZK03dYsPBCxyKtA01rwf4BQv+z26SuS6qozXMD7Q3uv1VAakbLWrBlGOSl1bRFHObgDfgQ5yeLV6tR6JLiqTFtffF4zDn61EQmTYTQAz0MingmqRX0t8Op6bwr7mb736mf3Z80ugLbTcUO4m8EPnjvtMy2Hl36HeaZjXL9kZq86logQKBzcNNDsnS7PqzTeBMLuaMHeZ5KVBcIzNNJAepxjA/QQ0+dLdye5E6FvypylypylnHw0NalxPS1nJndP7q5qFb1aWMwGybz2jMci7dPN4J+n8GYMylWUOVgK6/oltMkOpHpaTOKpQxZnZm1rD0a+ZJYVlh7EaNwTM= 2021-11-09T07:45:57.423 INFO:teuthology.kill:2021-11-09 07:45:57,423.423 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-09T07:45:57.428 INFO:teuthology.kill:2021-11-09 07:45:57,427.427 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-09T07:45:57.505 INFO:teuthology.kill:2021-11-09 07:45:57,505.505 INFO:teuthology.orchestra.console:Power off smithi072 2021-11-09T07:45:57.519 INFO:teuthology.kill:2021-11-09 07:45:57,518.518 INFO:teuthology.orchestra.console:Power off smithi142 2021-11-09T07:46:05.878 INFO:teuthology.kill:2021-11-09 07:46:05,877.877 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2021-11-09T07:46:05.887 INFO:teuthology.kill:2021-11-09 07:46:05,887.887 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-11-09T07:46:06.039 INFO:teuthology.kill:2021-11-09 07:46:06,038.038 INFO:teuthology.lock.ops:unlocked: smithi072.front.sepia.ceph.com 2021-11-09T07:46:06.043 INFO:teuthology.kill:2021-11-09 07:46:06,043.043 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com 2021-11-09T07:48:06.382 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-09T07:48:06.509 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi072 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-08_20:12:29-rados-wip-yuri4-testing-2021-11-08-1005-pacific-distro-basic-smithi/6491697 2021-11-09T07:48:06.510 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi142 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-08_20:12:29-rados-wip-yuri4-testing-2021-11-08-1005-pacific-distro-basic-smithi/6491697