2021-11-16T07:48:25.007 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-16T07:48:25.130 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging 2021-11-16T07:48:25.132 INFO:teuthology.lock.ops:Updating [ubuntu@smithi029.front.sepia.ceph.com]: reset os type and version on server 2021-11-16T07:48:25.133 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2021-11-16T07:48:25.154 INFO:teuthology.lock.ops:Node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging is complete 2021-11-16T07:48:25.154 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi158.front.sepia.ceph.com' reimaging 2021-11-16T07:48:25.154 INFO:teuthology.lock.ops:Updating [ubuntu@smithi158.front.sepia.ceph.com]: reset os type and version on server 2021-11-16T07:48:25.155 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2021-11-16T07:48:25.174 INFO:teuthology.lock.ops:Node 'ubuntu@smithi158.front.sepia.ceph.com' reimaging is complete 2021-11-16T07:48:25.388 INFO:teuthology.provision.fog.smithi158:Scheduling deploy of centos 8.stream 2021-11-16T07:48:25.391 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of centos 8.stream 2021-11-16T07:48:25.748 INFO:teuthology.orchestra.console:Power off smithi158 2021-11-16T07:48:25.763 INFO:teuthology.orchestra.console:Power off smithi029 2021-11-16T07:48:34.120 INFO:teuthology.orchestra.console:Power off for smithi158 completed 2021-11-16T07:48:34.125 INFO:teuthology.orchestra.console:Power off for smithi029 completed 2021-11-16T07:48:34.222 INFO:teuthology.orchestra.console:Power on smithi158 2021-11-16T07:48:34.232 INFO:teuthology.orchestra.console:Power on smithi029 2021-11-16T07:48:42.594 INFO:teuthology.orchestra.console:Power on for smithi158 completed 2021-11-16T07:48:42.599 INFO:teuthology.orchestra.console:Power on for smithi029 completed 2021-11-16T07:48:42.696 INFO:teuthology.provision.fog.smithi158:Waiting for deploy to finish 2021-11-16T07:48:42.700 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish 2021-11-16T07:53:47.504 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-16T07:53:47.842 INFO:teuthology.provision.fog.smithi029:Node is ready 2021-11-16T07:53:47.860 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029.front.sepia.ceph.com 2021-11-16T07:53:47.919 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29 smithi029.front.sepia.ceph.com smithi029 2021-11-16T07:53:48.275 INFO:teuthology.provision.fog.smithi029:Deploy complete! 2021-11-16T07:54:02.546 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-16T07:54:02.808 INFO:teuthology.provision.fog.smithi158:Node is ready 2021-11-16T07:54:02.865 INFO:teuthology.orchestra.run.smithi158.stdout:smithi158.front.sepia.ceph.com 2021-11-16T07:54:02.921 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158 smithi158.front.sepia.ceph.com smithi158 2021-11-16T07:54:03.262 INFO:teuthology.provision.fog.smithi158:Deploy complete! 2021-11-16T07:54:03.451 INFO:teuthology.lock.ops:Checking smithi029.front.sepia.ceph.com 2021-11-16T07:54:03.452 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-16T07:54:03.472 INFO:teuthology.lock.ops:Checking smithi158.front.sepia.ceph.com 2021-11-16T07:54:03.472 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-16T07:54:03.489 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: set os type and version on server 2021-11-16T07:54:03.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2021-11-16T07:54:03.737 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2021-11-16T07:54:03.795 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2021-11-16T07:54:03.796 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8" 2021-11-16T07:54:03.796 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2021-11-16T07:54:03.796 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2021-11-16T07:54:03.796 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8" 2021-11-16T07:54:03.797 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8" 2021-11-16T07:54:03.797 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-16T07:54:03.797 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2021-11-16T07:54:03.797 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-16T07:54:03.797 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2021-11-16T07:54:03.798 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-16T07:54:03.798 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-16T07:54:03.798 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-16T07:54:03.799 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2021-11-16T07:54:03.832 INFO:teuthology.lock.ops:Updating [smithi158.front.sepia.ceph.com]: set os type and version on server 2021-11-16T07:54:03.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2021-11-16T07:54:04.021 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2021-11-16T07:54:04.076 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="CentOS Stream" 2021-11-16T07:54:04.077 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="8" 2021-11-16T07:54:04.077 INFO:teuthology.orchestra.run.smithi158.stdout:ID="centos" 2021-11-16T07:54:04.077 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="rhel fedora" 2021-11-16T07:54:04.077 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="8" 2021-11-16T07:54:04.078 INFO:teuthology.orchestra.run.smithi158.stdout:PLATFORM_ID="platform:el8" 2021-11-16T07:54:04.078 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-16T07:54:04.078 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31" 2021-11-16T07:54:04.078 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-16T07:54:04.079 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://centos.org/" 2021-11-16T07:54:04.079 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-16T07:54:04.079 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-16T07:54:04.079 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-16T07:54:04.080 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2021-11-16T07:54:04.176 INFO:teuthology.dispatcher.supervisor:Running job 6505383 2021-11-16T07:54:04.176 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-16_00:15:25-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi/6505383 --name yuriw-2021-11-16_00:15:25-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2021-11-16_00:15:25-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi/6505383/orig.config.yaml 2021-11-16T07:54:04.180 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-11-16_00:15:25-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi/6505383 2021-11-16T07:54:04.181 INFO:teuthology.dispatcher.supervisor:Job PID: 1551 2021-11-16T07:54:04.181 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-16T19:54:58.407 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-16T19:54:58.609 INFO:teuthology.kill:Killing Pids: {1551} 2021-11-16T19:54:59.796 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-11-16T19:54:59.797 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-11-16T19:54:59.797 INFO:teuthology.misc:Compressing logs... 2021-11-16T19:54:59.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2021-11-16T19:54:59.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2021-11-16T19:55:00.342 INFO:teuthology.misc:Compressing logs... 2021-11-16T19:55:01.602 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-11-16T19:55:02.018 INFO:teuthology.misc:Compressing logs... 2021-11-16T19:55:15.542 INFO:teuthology.orchestra.run.smithi158.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-11-16T19:55:21.598 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-mon.smithi029.log: file size changed while zipping 2021-11-16T19:55:26.483 INFO:teuthology.orchestra.run.smithi158.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-11-16T19:55:27.304 INFO:teuthology.orchestra.run.smithi158.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.smithi158.jfeenb.log: file size changed while zipping 2021-11-16T19:55:33.845 INFO:teuthology.orchestra.run.smithi158.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-mgr.smithi158.efikcy.log: file size changed while zipping 2021-11-16T19:55:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-11-16T19:55:43.742 INFO:teuthology.orchestra.run.smithi158.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-mon.smithi158.log: file size changed while zipping 2021-11-16T19:55:44.931 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-11-16T19:55:54.272 INFO:teuthology.orchestra.run.smithi158.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-11-16T19:55:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/5e954ace-46b4-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-11-16T19:56:20.221 INFO:teuthology.kill:No teuthology processes running 2021-11-16T19:56:21.340 INFO:teuthology.kill:Nuking machines: ['smithi029', 'smithi158'] 2021-11-16T19:56:22.272 INFO:teuthology.kill:2021-11-16 19:56:22,272.272 INFO:teuthology.nuke:targets: 2021-11-16T19:56:22.273 INFO:teuthology.kill: smithi029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0nY0oCwe5/pRlyhoDh0ig52IN6Vz0bK26njJmNU0KXYtOtKviFm5uaAnhHQxiFZfD9nRGnF45iyuQoivgeJLfjkwrkYKHlyrPDS6iOpbh45eUzCo79kTkzel/POHL/L7e7jiOL6pj8OvGUHr6ik13XxF/6tXxgEKWjDSXYUGdf5CxpsZU2vHy0qgNlyxohGH4qDH763rJnSgLVne+7i5uraFO5j+huD6gFYudZEe95USImFmVryA4/1nYtOQFP9t3mYIJJjDNbfPb9izzMGGOuFJah80CL46SjZUsqR+75OjGolK+LDYKvGkqjBOXHlxUZ95I47hrvF4w2e8cv8AEzas5qyV4WCnwAQfMjX1YaQBblcU/PAZKa2dkgvl5m2MJeaDywxoIF0/vxNw+nPhuYu50cQPL94OFcFgNqbZGwKDlE2EyY/1JFzK9DG6Y9JBTbyCPJlkg6IoJSzO7Zt27QNpoN3+d80etN0ypXIth3YIvNVftkAVepXYofCE/Ubk= 2021-11-16T19:56:22.273 INFO:teuthology.kill: smithi158.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfeT9U2okHOcTkbCFxWDI87uZ39M5bDD0cNIsy0ZvxIMzKexXPU5h4cRjRwSahx/UvdIU1Ecmpm41BsG7zQoU4Djc3lXW2eVoJVA8cxn6n3d3bVxWzqURWr3f9itytwR/ynK44m48Tm6K9F7yeufXd4fpUxqLq4eZks4Oq4Kqlwj4pjp4xmko0yeIwSEDFDsMVZN2EfRVtet+RrDInGZEWrX1eBdx7R1tfG6C+Y5PYk+InsuOmk0bbL+uF49DAfSlZ/reUV/tpaDEzH+nWcwrw6tPKU2MCgBFVw5rvtAdh4KZ4TfS7qQbP0ImQXUalMigCA9R350EEH/bE/sUv6Rx3ie0IIEIfbXBmn+GVmZRtrc1FVTrBdluoc3DtazMEy/uS5XSPAAMXN0QfXghvdV62V+REB42bQ+ctLAanHBJuZdzxc6gWhcly8Ek5tFU5EMuIXgerNbnXA/ZxDLQ6LrQEPMi8FvBufbP0DbUQylFdkllmAbgJFBkTIfCyYiY1KHM= 2021-11-16T19:56:22.298 INFO:teuthology.kill:2021-11-16 19:56:22,298.298 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-16T19:56:22.301 INFO:teuthology.kill:2021-11-16 19:56:22,301.301 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-16T19:56:22.374 INFO:teuthology.kill:2021-11-16 19:56:22,374.374 INFO:teuthology.orchestra.console:Power off smithi029 2021-11-16T19:56:22.387 INFO:teuthology.kill:2021-11-16 19:56:22,387.387 INFO:teuthology.orchestra.console:Power off smithi158 2021-11-16T19:56:30.751 INFO:teuthology.kill:2021-11-16 19:56:30,751.751 INFO:teuthology.orchestra.console:Power off for smithi158 completed 2021-11-16T19:56:30.899 INFO:teuthology.kill:2021-11-16 19:56:30,899.899 INFO:teuthology.lock.ops:unlocked: smithi158.front.sepia.ceph.com 2021-11-16T19:56:34.859 INFO:teuthology.kill:2021-11-16 19:56:34,858.858 INFO:teuthology.orchestra.console:Power off for smithi029 completed 2021-11-16T19:56:35.004 INFO:teuthology.kill:2021-11-16 19:56:35,004.004 INFO:teuthology.lock.ops:unlocked: smithi029.front.sepia.ceph.com 2021-11-16T19:58:35.323 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-16T19:58:35.435 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi029 but it was locked by another job: /home/teuthworker/archive/swagner-2021-11-16_14:11:34-rados:cephadm-wip-swagner2-testing-2021-11-16-1128-pacific-distro-default-smithi/6506717 2021-11-16T19:58:35.436 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi158 but it was locked by another job: /home/teuthworker/archive/swagner-2021-11-16_14:11:34-rados:cephadm-wip-swagner2-testing-2021-11-16-1128-pacific-distro-default-smithi/6506718