2021-10-29T22:29:42.102 INFO:root:teuthology version: 1.1.0-c56135d1 2021-10-29T22:29:42.243 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging 2021-10-29T22:29:42.244 INFO:teuthology.lock.ops:Updating [ubuntu@smithi074.front.sepia.ceph.com]: reset os type and version on server 2021-10-29T22:29:42.245 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2021-10-29T22:29:42.267 INFO:teuthology.lock.ops:Node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging is complete 2021-10-29T22:29:42.267 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi119.front.sepia.ceph.com' reimaging 2021-10-29T22:29:42.267 INFO:teuthology.lock.ops:Updating [ubuntu@smithi119.front.sepia.ceph.com]: reset os type and version on server 2021-10-29T22:29:42.268 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2021-10-29T22:29:42.296 INFO:teuthology.lock.ops:Node 'ubuntu@smithi119.front.sepia.ceph.com' reimaging is complete 2021-10-29T22:29:42.484 INFO:teuthology.provision.fog.smithi074:Scheduling deploy of centos 8.3 2021-10-29T22:29:42.503 INFO:teuthology.provision.fog.smithi119:Scheduling deploy of centos 8.3 2021-10-29T22:29:42.828 INFO:teuthology.orchestra.console:Power off smithi074 2021-10-29T22:29:42.843 INFO:teuthology.orchestra.console:Power off smithi119 2021-10-29T22:29:42.979 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2021-10-29T22:29:42.989 INFO:teuthology.orchestra.console:Power off for smithi119 completed 2021-10-29T22:29:43.082 INFO:teuthology.orchestra.console:Power on smithi074 2021-10-29T22:29:43.089 INFO:teuthology.orchestra.console:Power on smithi119 2021-10-29T22:29:47.343 INFO:teuthology.orchestra.console:Power on for smithi119 completed 2021-10-29T22:29:47.444 INFO:teuthology.provision.fog.smithi119:Waiting for deploy to finish 2021-10-29T22:29:51.448 INFO:teuthology.orchestra.console:Power on for smithi074 completed 2021-10-29T22:29:51.549 INFO:teuthology.provision.fog.smithi074:Waiting for deploy to finish 2021-10-29T22:34:37.066 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-10-29T22:34:37.399 INFO:teuthology.provision.fog.smithi119:Node is ready 2021-10-29T22:34:37.413 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119.front.sepia.ceph.com 2021-10-29T22:34:37.468 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119 smithi119.front.sepia.ceph.com smithi119 2021-10-29T22:34:37.790 INFO:teuthology.provision.fog.smithi119:Deploy complete! 2021-10-29T22:34:40.901 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-10-29T22:34:41.189 INFO:teuthology.provision.fog.smithi074:Node is ready 2021-10-29T22:34:41.244 INFO:teuthology.orchestra.run.smithi074.stdout:smithi074.front.sepia.ceph.com 2021-10-29T22:34:41.299 INFO:teuthology.orchestra.run.smithi074.stdout:172.21.15.74 smithi074.front.sepia.ceph.com smithi074 2021-10-29T22:34:41.619 INFO:teuthology.provision.fog.smithi074:Deploy complete! 2021-10-29T22:34:41.770 INFO:teuthology.lock.ops:Checking smithi074.front.sepia.ceph.com 2021-10-29T22:34:41.771 INFO:teuthology.lock.ops:New key found. Updating... 2021-10-29T22:34:41.790 INFO:teuthology.lock.ops:Checking smithi119.front.sepia.ceph.com 2021-10-29T22:34:41.790 INFO:teuthology.lock.ops:New key found. Updating... 2021-10-29T22:34:41.808 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: set os type and version on server 2021-10-29T22:34:41.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com' 2021-10-29T22:34:42.066 INFO:teuthology.orchestra.run.smithi074.stdout:x86_64 2021-10-29T22:34:42.120 INFO:teuthology.orchestra.run.smithi074.stdout:NAME="CentOS Linux" 2021-10-29T22:34:42.121 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION="8" 2021-10-29T22:34:42.121 INFO:teuthology.orchestra.run.smithi074.stdout:ID="centos" 2021-10-29T22:34:42.121 INFO:teuthology.orchestra.run.smithi074.stdout:ID_LIKE="rhel fedora" 2021-10-29T22:34:42.121 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION_ID="8" 2021-10-29T22:34:42.122 INFO:teuthology.orchestra.run.smithi074.stdout:PLATFORM_ID="platform:el8" 2021-10-29T22:34:42.122 INFO:teuthology.orchestra.run.smithi074.stdout:PRETTY_NAME="CentOS Linux 8" 2021-10-29T22:34:42.122 INFO:teuthology.orchestra.run.smithi074.stdout:ANSI_COLOR="0;31" 2021-10-29T22:34:42.122 INFO:teuthology.orchestra.run.smithi074.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-29T22:34:42.123 INFO:teuthology.orchestra.run.smithi074.stdout:HOME_URL="https://centos.org/" 2021-10-29T22:34:42.123 INFO:teuthology.orchestra.run.smithi074.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-10-29T22:34:42.123 INFO:teuthology.orchestra.run.smithi074.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-10-29T22:34:42.123 INFO:teuthology.orchestra.run.smithi074.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-10-29T22:34:42.125 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2021-10-29T22:34:42.143 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: set os type and version on server 2021-10-29T22:34:42.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2021-10-29T22:34:42.408 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2021-10-29T22:34:42.462 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Linux" 2021-10-29T22:34:42.462 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="8" 2021-10-29T22:34:42.463 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2021-10-29T22:34:42.463 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2021-10-29T22:34:42.463 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="8" 2021-10-29T22:34:42.463 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el8" 2021-10-29T22:34:42.463 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Linux 8" 2021-10-29T22:34:42.464 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2021-10-29T22:34:42.464 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-29T22:34:42.464 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2021-10-29T22:34:42.464 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-10-29T22:34:42.465 INFO:teuthology.orchestra.run.smithi119.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-10-29T22:34:42.465 INFO:teuthology.orchestra.run.smithi119.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-10-29T22:34:42.465 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2021-10-29T22:34:42.564 INFO:teuthology.dispatcher.supervisor:Running job 6467498 2021-10-29T22:34:42.565 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi/6467498 --name yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi/6467498/orig.config.yaml 2021-10-29T22:34:42.568 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi/6467498 2021-10-29T22:34:42.569 INFO:teuthology.dispatcher.supervisor:Job PID: 6213 2021-10-29T22:34:42.569 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-10-30T10:35:37.635 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-10-30T10:35:38.090 INFO:teuthology.kill:Killing Pids: {6213} 2021-10-30T10:35:39.494 INFO:teuthology.task.internal:roles: ubuntu@smithi074.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-10-30T10:35:39.495 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-10-30T10:35:39.496 INFO:teuthology.misc:Compressing logs... 2021-10-30T10:35:39.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com' 2021-10-30T10:35:39.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2021-10-30T10:35:40.074 INFO:teuthology.misc:Compressing logs... 2021-10-30T10:35:40.429 INFO:teuthology.orchestra.run.smithi074.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-10-30T10:35:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-10-30T10:35:40.790 INFO:teuthology.misc:Compressing logs... 2021-10-30T10:36:02.923 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-10-30T10:36:09.955 INFO:teuthology.orchestra.run.smithi074.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-mon.smithi074.log: file size changed while zipping 2021-10-30T10:36:12.303 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-mds.cephfs.smithi119.zmunno.log: file size changed while zipping 2021-10-30T10:36:22.825 INFO:teuthology.orchestra.run.smithi074.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-mgr.smithi074.thuvfr.log: file size changed while zipping 2021-10-30T10:36:41.774 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-10-30T10:36:42.088 INFO:teuthology.orchestra.run.smithi074.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-10-30T10:37:01.381 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-10-30T10:37:07.651 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-mon.smithi119.log: file size changed while zipping 2021-10-30T10:37:08.366 INFO:teuthology.orchestra.run.smithi074.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-10-30T10:38:45.690 INFO:teuthology.orchestra.run.smithi074.stderr:gzip: /var/log/ceph/223689d8-390a-11ec-8c28-001a4aab830c/ceph-mds.cephfs.smithi074.rkgwck.log: file size changed while zipping 2021-10-30T10:39:44.124 INFO:teuthology.kill:No teuthology processes running 2021-10-30T10:39:45.276 INFO:teuthology.kill:Nuking machines: ['smithi074', 'smithi119'] 2021-10-30T10:39:46.271 INFO:teuthology.kill:2021-10-30 10:39:46,271.271 INFO:teuthology.nuke:targets: 2021-10-30T10:39:46.272 INFO:teuthology.kill: smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCV3p4pNarGRnAQPzNtdWSc8yYeRL9p7x9PO7pIGOzvi4CCl2z7g9KzEwF6Adi4Khm/BjRj9rkn/9pxW+yi8/0slv5+uViko7b6BQlHvNHPyWguM+ecSRbFeqVL4kVokzAoCzivEp9NiL3RXk+Vajf0PHoMmPoX6mqyVgzAPwj3GAeTd6R2YX/M0fvUj6vR7LspZDfXn5O1EX+QkleGnZA5kr4ABWu6OU+GTlvBhHnrYnifgjUkWabnAcWgvLOR3mpTM6z5zJf+86EdMxz75TXRBAM7e8BjgUi2sU0Isy/8tL1GZ3B1dtK4r6AZtpKI4ZxLZMt+JUdjB+K+krj1FBXegqUcKF3hufxW3XK2hq8fGP3auv3O/s9Vtn1/F8CrDq5Qc8Gdj6cUXbzFri3SR/QORuJRczZQiskmDCpO68yqIvpGfuz3YuZfziESxxvRBVLi63H8zGS34zXJy1DqJWi8Mya6CQfLC4nziaXKk/kf0SQ3virZkiZekC8O35sXtc= 2021-10-30T10:39:46.273 INFO:teuthology.kill: smithi119.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzyFXqU/GiW/+10zEoJH8e2H3asWU0U0xvNZHPIpNq6sWp+S6hlRTjyi1ZrQqP/ltpEXAS7cWkaPhWkz8+PCR7dCtj2w1QcBq8C6bQih62uCVL2iLNbo/mPj1nTnbpA5vx7F9FQIJZmjh/LjFamVWr9fE9IUh/iqZmZcw+TrzzVjFmDwQ/iv9hb2edXTOE2jxZRLd2WrPM/D4lW/nOefH10otECU65/S4PwnUh+pX/MkcCA1kEe+0ByP2vcsPyfISFGaYWlmjEOT9/E1B1mum6fsKZiRmPSrPQfMPuSheSMQD8wgjRhjo1Ubev2gfbCdQtUjDeyHxkDYv4RmVLHY++yoaMpfNjnXWdZ7ZCI/wCtQwEyoEg3GnyUPcl2egYAcwuxMVf3rqTmoSMNZ+LwqrMyQOAKcmvujreGtok/jCLrr1qbDWa4XSUpHfCRp1wWJDKUGqw5SFgQqSoir1+geqlxT5aWTzRJP+9F9oZiFvhdAUi0fHZKWrIHRWwCnpeNXc= 2021-10-30T10:39:46.300 INFO:teuthology.kill:2021-10-30 10:39:46,300.300 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-30T10:39:46.302 INFO:teuthology.kill:2021-10-30 10:39:46,302.302 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-30T10:39:46.378 INFO:teuthology.kill:2021-10-30 10:39:46,378.378 INFO:teuthology.orchestra.console:Power off smithi074 2021-10-30T10:39:46.391 INFO:teuthology.kill:2021-10-30 10:39:46,390.390 INFO:teuthology.orchestra.console:Power off smithi119 2021-10-30T10:39:58.857 INFO:teuthology.kill:2021-10-30 10:39:58,856.856 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2021-10-30T10:39:58.861 INFO:teuthology.kill:2021-10-30 10:39:58,861.861 INFO:teuthology.orchestra.console:Power off for smithi119 completed 2021-10-30T10:39:59.010 INFO:teuthology.kill:2021-10-30 10:39:59,009.009 INFO:teuthology.lock.ops:unlocked: smithi074.front.sepia.ceph.com 2021-10-30T10:39:59.014 INFO:teuthology.kill:2021-10-30 10:39:59,014.014 INFO:teuthology.lock.ops:unlocked: smithi119.front.sepia.ceph.com 2021-10-30T10:41:59.347 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-10-30T10:41:59.453 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi074 but it was locked by another job: /home/teuthworker/archive/adking-2021-10-30_00:36:30-rados:cephadm-wip-adk-testing-2021-10-29-1557-distro-basic-smithi/6468597 2021-10-30T10:41:59.453 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi119 but it was locked by another job: /home/teuthworker/archive/adking-2021-10-30_00:36:30-rados:cephadm-wip-adk-testing-2021-10-29-1557-distro-basic-smithi/6468597