2022-01-08T00:48:20.275 INFO:root:teuthology version: 1.1.0-95a7d479 2022-01-08T00:48:20.400 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi080.front.sepia.ceph.com' reimaging 2022-01-08T00:48:20.401 INFO:teuthology.lock.ops:Updating [ubuntu@smithi080.front.sepia.ceph.com]: reset os type and version on server 2022-01-08T00:48:20.402 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2022-01-08T00:48:20.442 INFO:teuthology.lock.ops:Node 'ubuntu@smithi080.front.sepia.ceph.com' reimaging is complete 2022-01-08T00:48:20.443 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi107.front.sepia.ceph.com' reimaging 2022-01-08T00:48:20.443 INFO:teuthology.lock.ops:Updating [ubuntu@smithi107.front.sepia.ceph.com]: reset os type and version on server 2022-01-08T00:48:20.443 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2022-01-08T00:48:20.472 INFO:teuthology.lock.ops:Node 'ubuntu@smithi107.front.sepia.ceph.com' reimaging is complete 2022-01-08T00:48:20.653 INFO:teuthology.provision.fog.smithi080:Scheduling deploy of centos 8.stream 2022-01-08T00:48:20.681 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of centos 8.stream 2022-01-08T00:48:21.085 INFO:teuthology.orchestra.console:Power off smithi107 2022-01-08T00:48:21.104 INFO:teuthology.orchestra.console:Power off smithi080 2022-01-08T00:48:21.244 INFO:teuthology.orchestra.console:Power off for smithi080 completed 2022-01-08T00:48:21.346 INFO:teuthology.orchestra.console:Power on smithi080 2022-01-08T00:48:25.604 INFO:teuthology.orchestra.console:Power on for smithi080 completed 2022-01-08T00:48:25.705 INFO:teuthology.provision.fog.smithi080:Waiting for deploy to finish 2022-01-08T00:48:33.577 INFO:teuthology.orchestra.console:Power off for smithi107 completed 2022-01-08T00:48:33.679 INFO:teuthology.orchestra.console:Power on smithi107 2022-01-08T00:48:37.933 INFO:teuthology.orchestra.console:Power on for smithi107 completed 2022-01-08T00:48:38.034 INFO:teuthology.provision.fog.smithi107:Waiting for deploy to finish 2022-01-08T00:53:43.332 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-08T00:53:43.609 INFO:teuthology.provision.fog.smithi107:Node is ready 2022-01-08T00:53:43.627 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107.front.sepia.ceph.com 2022-01-08T00:53:43.684 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107 smithi107.front.sepia.ceph.com smithi107 2022-01-08T00:53:44.030 INFO:teuthology.provision.fog.smithi107:Deploy complete! 2022-01-08T00:53:45.787 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-08T00:53:46.059 INFO:teuthology.provision.fog.smithi080:Node is ready 2022-01-08T00:53:46.077 INFO:teuthology.orchestra.run.smithi080.stdout:smithi080.front.sepia.ceph.com 2022-01-08T00:53:46.135 INFO:teuthology.orchestra.run.smithi080.stdout:172.21.15.80 smithi080.front.sepia.ceph.com smithi080 2022-01-08T00:53:46.495 INFO:teuthology.provision.fog.smithi080:Deploy complete! 2022-01-08T00:53:46.663 INFO:teuthology.lock.ops:Checking smithi080.front.sepia.ceph.com 2022-01-08T00:53:46.664 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-08T00:53:46.684 INFO:teuthology.lock.ops:Checking smithi107.front.sepia.ceph.com 2022-01-08T00:53:46.684 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-08T00:53:46.710 INFO:teuthology.lock.ops:Updating [smithi080.front.sepia.ceph.com]: set os type and version on server 2022-01-08T00:53:46.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2022-01-08T00:53:46.906 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2022-01-08T00:53:46.965 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="CentOS Stream" 2022-01-08T00:53:46.965 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="8" 2022-01-08T00:53:46.965 INFO:teuthology.orchestra.run.smithi080.stdout:ID="centos" 2022-01-08T00:53:46.966 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE="rhel fedora" 2022-01-08T00:53:46.966 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="8" 2022-01-08T00:53:46.966 INFO:teuthology.orchestra.run.smithi080.stdout:PLATFORM_ID="platform:el8" 2022-01-08T00:53:46.966 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="CentOS Stream 8" 2022-01-08T00:53:46.967 INFO:teuthology.orchestra.run.smithi080.stdout:ANSI_COLOR="0;31" 2022-01-08T00:53:46.967 INFO:teuthology.orchestra.run.smithi080.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-08T00:53:46.967 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://centos.org/" 2022-01-08T00:53:46.968 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-08T00:53:46.968 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-08T00:53:46.968 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-01-08T00:53:46.969 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2022-01-08T00:53:46.989 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: set os type and version on server 2022-01-08T00:53:46.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2022-01-08T00:53:47.183 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2022-01-08T00:53:47.241 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2022-01-08T00:53:47.242 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8" 2022-01-08T00:53:47.242 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2022-01-08T00:53:47.242 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2022-01-08T00:53:47.243 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8" 2022-01-08T00:53:47.243 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2022-01-08T00:53:47.243 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 8" 2022-01-08T00:53:47.244 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2022-01-08T00:53:47.244 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-08T00:53:47.244 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2022-01-08T00:53:47.244 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-08T00:53:47.245 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-08T00:53:47.245 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-01-08T00:53:47.246 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2022-01-08T00:53:47.345 INFO:teuthology.dispatcher.supervisor:Running job 6601681 2022-01-08T00:53:47.346 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-01-07_20:51:42-rados-wip-yuri-testing-2022-01-07-0928-pacific-distro-default-smithi/6601681 --name yuriw-2022-01-07_20:51:42-rados-wip-yuri-testing-2022-01-07-0928-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2022-01-07_20:51:42-rados-wip-yuri-testing-2022-01-07-0928-pacific-distro-default-smithi/6601681/orig.config.yaml 2022-01-08T00:53:47.349 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-01-07_20:51:42-rados-wip-yuri-testing-2022-01-07-0928-pacific-distro-default-smithi/6601681 2022-01-08T00:53:47.349 INFO:teuthology.dispatcher.supervisor:Job PID: 5701 2022-01-08T00:53:47.350 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-01-08T12:54:42.895 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-01-08T12:54:43.170 INFO:teuthology.kill:Killing Pids: {5701} 2022-01-08T12:54:44.338 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-01-08T12:54:44.339 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-01-08T12:54:44.339 INFO:teuthology.misc:Compressing logs... 2022-01-08T12:54:44.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2022-01-08T12:54:44.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2022-01-08T12:54:44.833 INFO:teuthology.misc:Compressing logs... 2022-01-08T12:54:46.056 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-01-08T12:54:46.471 INFO:teuthology.misc:Compressing logs... 2022-01-08T12:54:50.218 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-mds.cephfs.smithi080.szkrht.log: file size changed while zipping 2022-01-08T12:55:09.459 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-mon.smithi080.log: file size changed while zipping 2022-01-08T12:55:11.492 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-01-08T12:55:17.149 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-mgr.smithi080.xnglry.log: file size changed while zipping 2022-01-08T12:55:35.522 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-01-08T12:55:43.119 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-mds.cephfs.smithi107.quyvkt.log: file size changed while zipping 2022-01-08T12:55:48.756 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-mon.smithi107.log: file size changed while zipping 2022-01-08T12:55:59.503 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-01-08T12:56:10.495 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-01-08T12:56:31.292 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-mds.cephfs.smithi107.udqopk.log: file size changed while zipping 2022-01-08T12:56:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/098b8fd6-701f-11ec-8c32-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-01-08T12:57:28.533 INFO:teuthology.kill:No teuthology processes running 2022-01-08T12:57:29.701 INFO:teuthology.kill:Nuking machines: ['smithi080', 'smithi107'] 2022-01-08T12:57:30.692 INFO:teuthology.kill:2022-01-08 12:57:30,691.691 INFO:teuthology.nuke:targets: 2022-01-08T12:57:30.692 INFO:teuthology.kill: smithi080.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLO2wQzoAyC+jMEIj0mVY5W4E6sAj8eNZ1LsFZ7k8obQKB2kLfdmvnkLjlh/6cjIWPQDpkc5ZxblbkEr3S7YW3sGm6W5dvBFXQDtrVqlAj+bv0mp6hKGB4dEXONq9Pr/+n+9DO1OfICu3xSEvyywr4SjJIV8QQ2vwFYqq18db9arf8rESoNnhNayQvDRkb4ZgH9Qt9QcWXhDEiHWDtSUmDhOsClq6vGc7yOJCfpbMzMJ3vr5wjbhy+83mVSaLph75vIKDseQpprDT4nFMlFgE7XhLGCA+pW4KczOlPxkSm/+FelfN8+es3WZJ0CwEKAjC8qWysVKb89BQ6CiVXlw/ivbtTibqQ/iNkmZ8H7xZvGNbawCuQXFUtP060FK51w/mpLPxrl/o6ETRy/f1QwDkBg8dAv6V8tvI5AqK+ggzZhi8h22DOXHI27I+2PXJOeIGEeOQncfSQzW+FZjJ/Q/PrT57f4vFBOPAGghwxYoQl6uOAWK+axs2jrrMkdG4F8pc= 2022-01-08T12:57:30.693 INFO:teuthology.kill: smithi107.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVmd8O4xQqfWXNZrU4PK00ZtuNg5tykVIdLS+Tto5drrIKAgWiCU9ugWeT6acXsWmmh0PphdoUkszBbUTcBEYaDIf+LG2LszY3BqqXRNxzRt2X38lwGd0rj3CXHI7ODMDTlyRyDMETWtg1vbzTPcqnE09VuCARabDEsBTl8MsV2oAW9FyUSV5o596I2n8v8B8jbKep1Y2nfM6V3QPh3D1Ye0OiEEbXn6gpoWfhXkECGKE/8WHaGk2B6ZsBPcqoqm/MxlhrW4Y/KMNUSA/cEXdH3dRqisxMDrzakBRXr6d737pMwrwxxrVqj/SVrA578+aC8E2TecNUG+e+7lPzSzokXi/dPkOm6l/4++4ENJG1tSubiG/Pi/XiJjUSCs3tt1LtUn2MnDsGM+3CThgJRbgFxsjJLc+ZU1IBZJJtNBMF+RT80HWPooxkZsBLOzyTMD0h4FrgGizXvsfpfeK5hUSenqwB668iJ8UqC6MLUWk+p3t/qHaNhuM53CTK2cIunWc= 2022-01-08T12:57:30.722 INFO:teuthology.kill:2022-01-08 12:57:30,721.721 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-08T12:57:30.725 INFO:teuthology.kill:2022-01-08 12:57:30,725.725 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-08T12:57:30.804 INFO:teuthology.kill:2022-01-08 12:57:30,803.803 INFO:teuthology.orchestra.console:Power off smithi080 2022-01-08T12:57:30.817 INFO:teuthology.kill:2022-01-08 12:57:30,817.817 INFO:teuthology.orchestra.console:Power off smithi107 2022-01-08T12:57:39.180 INFO:teuthology.kill:2022-01-08 12:57:39,179.179 INFO:teuthology.orchestra.console:Power off for smithi107 completed 2022-01-08T12:57:39.326 INFO:teuthology.kill:2022-01-08 12:57:39,326.326 INFO:teuthology.lock.ops:unlocked: smithi107.front.sepia.ceph.com 2022-01-08T12:57:43.285 INFO:teuthology.kill:2022-01-08 12:57:43,284.284 INFO:teuthology.orchestra.console:Power off for smithi080 completed 2022-01-08T12:57:43.435 INFO:teuthology.kill:2022-01-08 12:57:43,435.435 INFO:teuthology.lock.ops:unlocked: smithi080.front.sepia.ceph.com 2022-01-08T12:59:43.753 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-01-08T12:59:43.867 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi080 but it was locked by another job: /home/teuthworker/archive/dis-2022-01-08_10:51:32-rbd-wip-dis-testing-distro-default-smithi/6602262 2022-01-08T12:59:43.868 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi107 but it was locked by another job: /home/teuthworker/archive/dis-2022-01-08_10:51:32-rbd-wip-dis-testing-distro-default-smithi/6602261