2022-02-24T22:37:00.262 INFO:root:teuthology version: 1.1.0-5825e8d0
2022-02-24T22:37:00.381 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging
2022-02-24T22:37:00.382 INFO:teuthology.lock.ops:Updating [ubuntu@smithi140.front.sepia.ceph.com]: reset os type and version on server
2022-02-24T22:37:00.382 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2022-02-24T22:37:00.400 INFO:teuthology.lock.ops:Node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging is complete
2022-02-24T22:37:00.401 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi157.front.sepia.ceph.com' reimaging
2022-02-24T22:37:00.401 INFO:teuthology.lock.ops:Updating [ubuntu@smithi157.front.sepia.ceph.com]: reset os type and version on server
2022-02-24T22:37:00.401 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server
2022-02-24T22:37:00.420 INFO:teuthology.lock.ops:Node 'ubuntu@smithi157.front.sepia.ceph.com' reimaging is complete
2022-02-24T22:37:00.614 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of centos 8.stream
2022-02-24T22:37:00.650 INFO:teuthology.provision.fog.smithi157:Scheduling deploy of centos 8.stream
2022-02-24T22:37:00.968 INFO:teuthology.orchestra.console:Power off smithi140
2022-02-24T22:37:01.018 INFO:teuthology.orchestra.console:Power off smithi157
2022-02-24T22:37:01.108 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2022-02-24T22:37:01.160 INFO:teuthology.orchestra.console:Power off for smithi157 completed
2022-02-24T22:37:01.209 INFO:teuthology.orchestra.console:Power on smithi140
2022-02-24T22:37:01.261 INFO:teuthology.orchestra.console:Power on smithi157
2022-02-24T22:37:05.465 INFO:teuthology.orchestra.console:Power on for smithi140 completed
2022-02-24T22:37:05.518 INFO:teuthology.orchestra.console:Power on for smithi157 completed
2022-02-24T22:37:05.567 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2022-02-24T22:37:05.619 INFO:teuthology.provision.fog.smithi157:Waiting for deploy to finish
2022-02-24T22:42:01.628 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-24T22:42:01.899 INFO:teuthology.provision.fog.smithi140:Node is ready
2022-02-24T22:42:01.917 INFO:teuthology.orchestra.run.smithi140.stdout:smithi140.front.sepia.ceph.com
2022-02-24T22:42:01.974 INFO:teuthology.orchestra.run.smithi140.stdout:172.21.15.140 smithi140.front.sepia.ceph.com smithi140
2022-02-24T22:42:02.327 INFO:teuthology.provision.fog.smithi140:Deploy complete!
2022-02-24T22:42:16.683 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-24T22:42:16.940 INFO:teuthology.provision.fog.smithi157:Node is ready
2022-02-24T22:42:16.958 INFO:teuthology.orchestra.run.smithi157.stdout:smithi157.front.sepia.ceph.com
2022-02-24T22:42:17.016 INFO:teuthology.orchestra.run.smithi157.stdout:172.21.15.157 smithi157.front.sepia.ceph.com smithi157
2022-02-24T22:42:17.370 INFO:teuthology.provision.fog.smithi157:Deploy complete!
2022-02-24T22:42:17.539 INFO:teuthology.lock.ops:Checking smithi140.front.sepia.ceph.com
2022-02-24T22:42:17.540 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-24T22:42:17.560 INFO:teuthology.lock.ops:Checking smithi157.front.sepia.ceph.com
2022-02-24T22:42:17.560 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-24T22:42:17.580 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: set os type and version on server
2022-02-24T22:42:17.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2022-02-24T22:42:17.828 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64
2022-02-24T22:42:17.887 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="CentOS Stream"
2022-02-24T22:42:17.888 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="8"
2022-02-24T22:42:17.888 INFO:teuthology.orchestra.run.smithi140.stdout:ID="centos"
2022-02-24T22:42:17.888 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE="rhel fedora"
2022-02-24T22:42:17.888 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="8"
2022-02-24T22:42:17.888 INFO:teuthology.orchestra.run.smithi140.stdout:PLATFORM_ID="platform:el8"
2022-02-24T22:42:17.889 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="CentOS Stream 8"
2022-02-24T22:42:17.889 INFO:teuthology.orchestra.run.smithi140.stdout:ANSI_COLOR="0;31"
2022-02-24T22:42:17.889 INFO:teuthology.orchestra.run.smithi140.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-02-24T22:42:17.889 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://centos.org/"
2022-02-24T22:42:17.890 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-24T22:42:17.890 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-24T22:42:17.890 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-02-24T22:42:17.892 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2022-02-24T22:42:17.910 INFO:teuthology.lock.ops:Updating [smithi157.front.sepia.ceph.com]: set os type and version on server
2022-02-24T22:42:17.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com'
2022-02-24T22:42:18.103 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64
2022-02-24T22:42:18.160 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream"
2022-02-24T22:42:18.160 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="8"
2022-02-24T22:42:18.160 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos"
2022-02-24T22:42:18.161 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora"
2022-02-24T22:42:18.161 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="8"
2022-02-24T22:42:18.161 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el8"
2022-02-24T22:42:18.161 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 8"
2022-02-24T22:42:18.162 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31"
2022-02-24T22:42:18.162 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-02-24T22:42:18.162 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/"
2022-02-24T22:42:18.162 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-24T22:42:18.163 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-24T22:42:18.163 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-02-24T22:42:18.163 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server
2022-02-24T22:42:18.263 INFO:teuthology.dispatcher.supervisor:Running job 6704776
2022-02-24T22:42:18.263 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_5825e8d094b6aff4351d26f1ceb5efc598c156a0/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-02-24_22:04:22-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6704776 --name yuriw-2022-02-24_22:04:22-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2022-02-24_22:04:22-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6704776/orig.config.yaml
2022-02-24T22:42:18.267 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-02-24_22:04:22-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6704776
2022-02-24T22:42:18.267 INFO:teuthology.dispatcher.supervisor:Job PID: 20849
2022-02-24T22:42:18.268 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-25T05:12:46.736 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-25T05:12:46.936 INFO:teuthology.kill:Killing Pids: {20849}
2022-02-25T05:12:48.068 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-02-25T05:12:48.069 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-02-25T05:12:48.069 INFO:teuthology.misc:Compressing logs...
2022-02-25T05:12:48.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2022-02-25T05:12:48.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com'
2022-02-25T05:12:48.307 INFO:teuthology.orchestra.run.smithi140.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-02-25T05:12:48.475 INFO:teuthology.orchestra.run.smithi157.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-02-25T05:12:48.519 INFO:teuthology.orchestra.run.smithi140.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-02-25T05:12:48.519 INFO:teuthology.orchestra.run.smithi140.stderr:tar: Error is not recoverable: exiting now
2022-02-25T05:12:48.566 INFO:teuthology.orchestra.run.smithi157.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-02-25T05:12:48.567 INFO:teuthology.orchestra.run.smithi157.stderr:tar: Error is not recoverable: exiting now
2022-02-25T05:12:48.568 INFO:teuthology.misc:Compressing logs...
2022-02-25T05:12:48.979 INFO:teuthology.misc:Compressing logs...
2022-02-25T05:12:50.079 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-mon.smithi157.log: file size changed while zipping
2022-02-25T05:12:52.630 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-mon.smithi140.log: file size changed while zipping
2022-02-25T05:12:52.938 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-02-25T05:12:55.454 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-02-25T05:12:56.183 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-02-25T05:12:57.774 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-02-25T05:12:58.858 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-mgr.smithi140.ivwtzx.log: file size changed while zipping
2022-02-25T05:12:58.882 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-02-25T05:13:02.136 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-02-25T05:13:02.282 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-02-25T05:13:05.987 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/2e235de0-95c4-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-02-25T05:13:49.841 INFO:teuthology.orchestra.run.smithi157.stderr:tar: .: file changed as we read it
2022-02-25T05:13:49.907 INFO:teuthology.kill:No teuthology processes running
2022-02-25T05:13:51.082 INFO:teuthology.kill:Nuking machines: ['smithi140', 'smithi157']
2022-02-25T05:13:52.045 INFO:teuthology.kill:2022-02-25 05:13:52,045.045 INFO:teuthology.nuke:targets:
2022-02-25T05:13:52.046 INFO:teuthology.kill:  smithi140.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAiOA0O9r2c9wY2PDMGwNdv4PEAuRc1gkavRDSeUkd7U9MIsHNt9VWhSqYXXCdMhpF9puOueLp4CoSaApKGx/+kAPal3ryZ5DeYQkVlZWxjl90Q3e3i6xUK2xjtNQ7MrvQTbqY9FRMcjXf31LxXOV813OH4U/t52Dy9+H0avd5cnKxXw5Tv+YHALiTWFYNlEms3U6g+KiSn+N0s4bq0fDT8LJfNsNnsPuvW64IbDRFqRF1on3xPY45/jWtAsA77asNVK6WTEj6AtRJnUGDTSsovI3n5IwJsihUY5cURx0jJhS88eFazjEgl1Ya7tLNMGtCecTrQ0tq7a3e9p5zy3ea0EY9p2FEHrz9ksG2Fjo+jW7QTe48sjPbfaHHkp71C5D64ssQ2i2bekjgupG48KFygmrTJdstu1fmdIjnIRNUVzD4LsWDQQiEB3dRJjTDxoo4hQs2bCF21xDGgi3IHBJU5W/qQ5kBVKmFJToa6Tlvd0o7f45TzQZvajjRYWGxNfM=
2022-02-25T05:13:52.047 INFO:teuthology.kill:  smithi157.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfDjuiwdAf3CaBpdBG816gRWIJb7T5TtD6qrph9w0owSz5aITcJUOMDeGXRh3sFBWC87PAWydQkgi8UsxAw3UHa5w82NjVgN0WhrQEmvZ7R/h+fZibvAt6d2PFnI6LwM9h6Z6SHAYdT93/kRwmVirnM1WvuYJvMYws6TNjJCFzBgP/+nd62Pu2BNbp5tcxQRPt3jPNnmvihIAdEdHCRpwVZV2NrC0GkB4iSoQ7Vy8+5sHioWTCWMmC0JqScnCZKWEFOpeTaXtCfeSfpzbFVJdYnHRDsVd6Tq2ppIM7bPuO6VAEuRouiz+qZzQaW4VgBT5L8D3275wjsiy3kD1rsR3brD5kMMWiEHGF5jXmhCZ9j0b9v2kDdfHEEWdylgz40uPOpxironIYfk8syTc1FQpCw8vKviL8+YrNSqP0A0RGEHLavyijg4ZyWC6UbP9gO3JUv6JBqycWALZYlzydcRi5zTnR0H1GNG1hYR119hiL7dFowK8RjvypT8wmsjjopu8=
2022-02-25T05:13:52.073 INFO:teuthology.kill:2022-02-25 05:13:52,073.073 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-25T05:13:52.077 INFO:teuthology.kill:2022-02-25 05:13:52,077.077 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-25T05:13:52.154 INFO:teuthology.kill:2022-02-25 05:13:52,154.154 INFO:teuthology.orchestra.console:Power off smithi140
2022-02-25T05:13:52.168 INFO:teuthology.kill:2022-02-25 05:13:52,168.168 INFO:teuthology.orchestra.console:Power off smithi157
2022-02-25T05:14:00.526 INFO:teuthology.kill:2022-02-25 05:14:00,525.525 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2022-02-25T05:14:00.531 INFO:teuthology.kill:2022-02-25 05:14:00,531.531 INFO:teuthology.orchestra.console:Power off for smithi157 completed
2022-02-25T05:14:00.681 INFO:teuthology.kill:2022-02-25 05:14:00,680.680 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2022-02-25T05:14:00.682 INFO:teuthology.kill:2022-02-25 05:14:00,681.681 INFO:teuthology.lock.ops:unlocked: smithi157.front.sepia.ceph.com
2022-02-25T05:16:00.991 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-25T05:16:01.090 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi140 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-02-18_04:10:03-fs-octopus-distro-default-smithi/6693742
2022-02-25T05:16:01.090 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi157 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-02-18_04:10:03-fs-octopus-distro-default-smithi/6693742