2021-11-11T10:59:02.755 INFO:root:teuthology version: 1.1.0-91815aeb
2021-11-11T10:59:02.881 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging
2021-11-11T10:59:02.882 INFO:teuthology.lock.ops:Updating [ubuntu@smithi140.front.sepia.ceph.com]: reset os type and version on server
2021-11-11T10:59:02.883 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2021-11-11T10:59:02.902 INFO:teuthology.lock.ops:Node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging is complete
2021-11-11T10:59:02.902 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging
2021-11-11T10:59:02.903 INFO:teuthology.lock.ops:Updating [ubuntu@smithi142.front.sepia.ceph.com]: reset os type and version on server
2021-11-11T10:59:02.903 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2021-11-11T10:59:02.924 INFO:teuthology.lock.ops:Node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging is complete
2021-11-11T10:59:03.119 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of centos 8.3
2021-11-11T10:59:03.159 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of centos 8.3
2021-11-11T10:59:03.457 INFO:teuthology.orchestra.console:Power off smithi140
2021-11-11T10:59:03.516 INFO:teuthology.orchestra.console:Power off smithi142
2021-11-11T10:59:11.831 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2021-11-11T10:59:11.932 INFO:teuthology.orchestra.console:Power on smithi140
2021-11-11T10:59:15.999 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2021-11-11T10:59:16.101 INFO:teuthology.orchestra.console:Power on smithi142
2021-11-11T10:59:20.301 INFO:teuthology.orchestra.console:Power on for smithi140 completed
2021-11-11T10:59:20.403 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2021-11-11T10:59:24.467 INFO:teuthology.orchestra.console:Power on for smithi142 completed
2021-11-11T10:59:24.568 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish
2021-11-11T11:01:51.401 ERROR:teuthology.orchestra.connection:Error authenticating with smithi140.front.sepia.ceph.com: Authentication failed.
2021-11-11T11:04:04.781 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-11T11:04:05.063 INFO:teuthology.provision.fog.smithi140:Node is ready
2021-11-11T11:04:05.119 INFO:teuthology.orchestra.run.smithi140.stdout:smithi140.front.sepia.ceph.com
2021-11-11T11:04:05.176 INFO:teuthology.orchestra.run.smithi140.stdout:172.21.15.140 smithi140.front.sepia.ceph.com smithi140
2021-11-11T11:04:05.499 INFO:teuthology.provision.fog.smithi140:Deploy complete!
2021-11-11T11:04:13.818 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-11T11:04:14.103 INFO:teuthology.provision.fog.smithi142:Node is ready
2021-11-11T11:04:14.117 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com
2021-11-11T11:04:14.174 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142
2021-11-11T11:04:14.494 INFO:teuthology.provision.fog.smithi142:Deploy complete!
2021-11-11T11:04:14.679 INFO:teuthology.lock.ops:Checking smithi140.front.sepia.ceph.com
2021-11-11T11:04:14.680 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-11T11:04:14.699 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com
2021-11-11T11:04:14.699 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-11T11:04:14.717 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: set os type and version on server
2021-11-11T11:04:14.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2021-11-11T11:04:14.983 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64
2021-11-11T11:04:15.038 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="CentOS Linux"
2021-11-11T11:04:15.038 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="8"
2021-11-11T11:04:15.039 INFO:teuthology.orchestra.run.smithi140.stdout:ID="centos"
2021-11-11T11:04:15.039 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE="rhel fedora"
2021-11-11T11:04:15.039 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="8"
2021-11-11T11:04:15.039 INFO:teuthology.orchestra.run.smithi140.stdout:PLATFORM_ID="platform:el8"
2021-11-11T11:04:15.040 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-11T11:04:15.040 INFO:teuthology.orchestra.run.smithi140.stdout:ANSI_COLOR="0;31"
2021-11-11T11:04:15.040 INFO:teuthology.orchestra.run.smithi140.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-11T11:04:15.040 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://centos.org/"
2021-11-11T11:04:15.041 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-11T11:04:15.041 INFO:teuthology.orchestra.run.smithi140.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-11T11:04:15.041 INFO:teuthology.orchestra.run.smithi140.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-11T11:04:15.042 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2021-11-11T11:04:15.061 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server
2021-11-11T11:04:15.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2021-11-11T11:04:15.265 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64
2021-11-11T11:04:15.319 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Linux"
2021-11-11T11:04:15.319 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8"
2021-11-11T11:04:15.320 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos"
2021-11-11T11:04:15.320 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora"
2021-11-11T11:04:15.320 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8"
2021-11-11T11:04:15.320 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8"
2021-11-11T11:04:15.321 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-11T11:04:15.321 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31"
2021-11-11T11:04:15.321 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-11T11:04:15.321 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/"
2021-11-11T11:04:15.322 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-11T11:04:15.322 INFO:teuthology.orchestra.run.smithi142.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-11T11:04:15.322 INFO:teuthology.orchestra.run.smithi142.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-11T11:04:15.323 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2021-11-11T11:04:15.418 INFO:teuthology.dispatcher.supervisor:Running job 6497199
2021-11-11T11:04:15.419 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_91815aeb94449f1ea4365867a0fd4c60103057e1/virtualenv/bin/teuthology -v --owner scheduled_swagner@teuthology --archive /home/teuthworker/archive/swagner-2021-11-11_10:56:08-orch:cephadm-wip-swagner-testing-2021-11-10-1442-distro-default-smithi/6497199 --name swagner-2021-11-11_10:56:08-orch:cephadm-wip-swagner-testing-2021-11-10-1442-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/swagner-2021-11-11_10:56:08-orch:cephadm-wip-swagner-testing-2021-11-10-1442-distro-default-smithi/6497199/orig.config.yaml
2021-11-11T11:04:15.422 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/swagner-2021-11-11_10:56:08-orch:cephadm-wip-swagner-testing-2021-11-10-1442-distro-default-smithi/6497199
2021-11-11T11:04:15.423 INFO:teuthology.dispatcher.supervisor:Job PID: 18055
2021-11-11T11:04:15.423 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-11T23:04:35.719 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-11T23:04:35.915 INFO:teuthology.kill:Killing Pids: {18055}
2021-11-11T23:04:37.814 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-11T23:04:37.815 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-11T23:04:37.815 INFO:teuthology.misc:Compressing logs...
2021-11-11T23:04:37.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2021-11-11T23:04:38.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2021-11-11T23:04:38.331 INFO:teuthology.misc:Compressing logs...
2021-11-11T23:04:38.622 INFO:teuthology.misc:Compressing logs...
2021-11-11T23:04:47.269 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-mgr.smithi140.ikbkhq.log: file size changed while zipping
2021-11-11T23:04:52.913 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-11-11T23:04:53.058 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-mgr.smithi142.fhnogf.log: file size changed while zipping
2021-11-11T23:05:07.571 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-mon.smithi140.log: file size changed while zipping
2021-11-11T23:05:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-11-11T23:05:10.849 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.smithi140.nvbodo.log: file size changed while zipping
2021-11-11T23:05:23.771 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-11-11T23:05:23.864 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.smithi142.mqesox.log: file size changed while zipping
2021-11-11T23:05:25.259 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2021-11-11T23:05:30.253 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-mon.smithi142.log: file size changed while zipping
2021-11-11T23:06:06.393 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.smithi140.mcnfnn.log: file size changed while zipping
2021-11-11T23:06:19.350 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-11-11T23:06:34.596 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/c708d33c-42e0-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-11-11T23:07:00.282 INFO:teuthology.orchestra.run.smithi140.stderr:tar: .: file changed as we read it
2021-11-11T23:07:10.666 INFO:teuthology.kill:No teuthology processes running
2021-11-11T23:07:11.766 INFO:teuthology.kill:Nuking machines: ['smithi140', 'smithi142']
2021-11-11T23:07:12.681 INFO:teuthology.kill:2021-11-11 23:07:12,681.681 INFO:teuthology.nuke:targets:
2021-11-11T23:07:12.682 INFO:teuthology.kill:  smithi140.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEmORoBYbXms7F3eb7P0boRVphwroswdVH8u8SKKwz4fTtSecc9XQQ/2ApKT3YDIaSlG+r3kW0Wc06ueZc7IPMBCyRBZ+TwWUbkB9m6RoUY5KV3CDOvGiiP8Jpg3sNlnxpA6dJ/tJQu6kNY34kA63T4kpN3YEQv56ruYSMzw00+63MNOfkeVaCtfGchhBohqyKm8TFGN+TKESoj/w01Q7EqA854lWv6fkf27PAf7VTReAfAehXOezFIFzUC575WEBBYYk8x1/riwEHer2tWCb4NGkgutKrg9LLZm1Riyz2B+AymVdrgcCJrg8pK6ODf73rh619ZZbmiD8YiLUWHP0y3SN3RzIcEKN58jCa0TGdOOrVwmq7D1Omt9X+AcocfMlx4VmcRjH6VDGM3kOPYRfe0zWVwaqQCj5weg8bZkqHWKiHvxC3RhxxJDzHkmy1AZmTZCD5yc57NTShCtOIzCS+xkZn6vwwvvzsGBGCyn8+sHmpGbibFgE+zwO2gaQTokc=
2021-11-11T23:07:12.683 INFO:teuthology.kill:  smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDyuIfcSivZzdY0rxNZP5tiELyXs8YDkgJDPY4Q9TnTkFEleiqued7kDXjkM0BKJRB6UIUe4s3PSjDEXBWBHwU+/Rjj61MWlFdATkFIUDh+gUXgDzV8yhtUN5ZwGJqBsjep0FUUYtrtTUgjprHn20LKgzHSgV1uLOCMvxaRlqmr/F0UGy+M/bE61FnZFsfgSOp99nnUFy8eWf+ZmP3Ib8s7w9zV0mYnj/rNoTgQ0N9PzJEl+UNgT+w4Odvb+GLor5tTvGb9kjMVKXNGPRvg30QIOh0Ji1xaQMlUNaY/FGIh/WVJiDpe3oFXsONqSfF2r4SbgwvXTJT5PEdavp9j19gQBnopQ7mdX5Z8PsGGTlhI3iOjidiuM/KOCxTCKzIWjrD+xR0CKLXc5Bk4n8ke7xDo4KfyyVMie2HiP9nSdA16i63Ui1C13Jf0ZqAyKof4GfLmicWtWK5Y5+sh26z/FAxKYA4kN6hkz4G+DDUX6YR9ifjY9g15w5HG+JU6EHL5tU=
2021-11-11T23:07:12.709 INFO:teuthology.kill:2021-11-11 23:07:12,708.708 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T23:07:12.712 INFO:teuthology.kill:2021-11-11 23:07:12,712.712 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T23:07:12.786 INFO:teuthology.kill:2021-11-11 23:07:12,786.786 INFO:teuthology.orchestra.console:Power off smithi140
2021-11-11T23:07:12.800 INFO:teuthology.kill:2021-11-11 23:07:12,799.799 INFO:teuthology.orchestra.console:Power off smithi142
2021-11-11T23:07:21.165 INFO:teuthology.kill:2021-11-11 23:07:21,165.165 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2021-11-11T23:07:21.310 INFO:teuthology.kill:2021-11-11 23:07:21,309.309 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com
2021-11-11T23:07:25.273 INFO:teuthology.kill:2021-11-11 23:07:25,272.272 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2021-11-11T23:07:25.418 INFO:teuthology.kill:2021-11-11 23:07:25,418.418 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2021-11-11T23:09:25.611 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-11T23:09:25.721 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi140 but it was locked by another job: /home/teuthworker/archive/benhanokh-2021-11-10_07:27:45-rados-restore_alloc_file-distro-basic-smithi/6495554
2021-11-11T23:09:25.721 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi142 but it was locked by another job: /home/teuthworker/archive/benhanokh-2021-11-10_07:27:45-rados-restore_alloc_file-distro-basic-smithi/6495554