2022-03-02T03:01:09.695 INFO:root:teuthology version: 1.1.0-cb5b1a75
2022-03-02T03:01:09.844 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging
2022-03-02T03:01:09.845 INFO:teuthology.lock.ops:Updating [ubuntu@smithi142.front.sepia.ceph.com]: reset os type and version on server
2022-03-02T03:01:09.846 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2022-03-02T03:01:09.868 INFO:teuthology.lock.ops:Node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging is complete
2022-03-02T03:01:09.869 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi150.front.sepia.ceph.com' reimaging
2022-03-02T03:01:09.869 INFO:teuthology.lock.ops:Updating [ubuntu@smithi150.front.sepia.ceph.com]: reset os type and version on server
2022-03-02T03:01:09.869 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server
2022-03-02T03:01:09.908 INFO:teuthology.lock.ops:Node 'ubuntu@smithi150.front.sepia.ceph.com' reimaging is complete
2022-03-02T03:01:10.112 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of centos 8.stream
2022-03-02T03:01:10.124 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of centos 8.stream
2022-03-02T03:01:10.531 INFO:teuthology.orchestra.console:Power off smithi142
2022-03-02T03:01:10.592 INFO:teuthology.orchestra.console:Power off smithi150
2022-03-02T03:01:18.903 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2022-03-02T03:01:18.956 INFO:teuthology.orchestra.console:Power off for smithi150 completed
2022-03-02T03:01:19.005 INFO:teuthology.orchestra.console:Power on smithi142
2022-03-02T03:01:19.058 INFO:teuthology.orchestra.console:Power on smithi150
2022-03-02T03:01:27.376 INFO:teuthology.orchestra.console:Power on for smithi142 completed
2022-03-02T03:01:27.424 INFO:teuthology.orchestra.console:Power on for smithi150 completed
2022-03-02T03:01:27.478 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish
2022-03-02T03:01:27.526 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish
2022-03-02T03:05:51.202 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-02T03:05:51.461 INFO:teuthology.provision.fog.smithi142:Node is ready
2022-03-02T03:05:51.479 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com
2022-03-02T03:05:51.537 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142
2022-03-02T03:05:51.882 INFO:teuthology.provision.fog.smithi142:Deploy complete!
2022-03-02T03:06:24.178 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-02T03:06:24.459 INFO:teuthology.provision.fog.smithi150:Node is ready
2022-03-02T03:06:24.477 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150.front.sepia.ceph.com
2022-03-02T03:06:24.534 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150
2022-03-02T03:06:24.876 INFO:teuthology.provision.fog.smithi150:Deploy complete!
2022-03-02T03:06:25.024 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com
2022-03-02T03:06:25.025 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-02T03:06:25.045 INFO:teuthology.lock.ops:Checking smithi150.front.sepia.ceph.com
2022-03-02T03:06:25.046 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-02T03:06:25.065 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server
2022-03-02T03:06:25.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2022-03-02T03:06:25.309 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64
2022-03-02T03:06:25.366 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream"
2022-03-02T03:06:25.366 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8"
2022-03-02T03:06:25.366 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos"
2022-03-02T03:06:25.367 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora"
2022-03-02T03:06:25.367 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8"
2022-03-02T03:06:25.367 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8"
2022-03-02T03:06:25.367 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-02T03:06:25.368 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31"
2022-03-02T03:06:25.368 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-02T03:06:25.368 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/"
2022-03-02T03:06:25.368 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-02T03:06:25.369 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-02T03:06:25.369 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-02T03:06:25.371 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2022-03-02T03:06:25.390 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: set os type and version on server
2022-03-02T03:06:25.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com'
2022-03-02T03:06:25.576 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64
2022-03-02T03:06:25.633 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream"
2022-03-02T03:06:25.634 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8"
2022-03-02T03:06:25.634 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos"
2022-03-02T03:06:25.634 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora"
2022-03-02T03:06:25.635 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8"
2022-03-02T03:06:25.635 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8"
2022-03-02T03:06:25.635 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-02T03:06:25.635 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31"
2022-03-02T03:06:25.635 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-02T03:06:25.636 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/"
2022-03-02T03:06:25.636 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-02T03:06:25.636 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-02T03:06:25.636 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-02T03:06:25.637 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server
2022-03-02T03:06:25.735 INFO:teuthology.dispatcher.supervisor:Running job 6714778
2022-03-02T03:06:25.735 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-03-01_17:45:51-rados-wip-yuri3-testing-2022-02-28-0757-pacific-distro-default-smithi/6714778 --name yuriw-2022-03-01_17:45:51-rados-wip-yuri3-testing-2022-02-28-0757-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2022-03-01_17:45:51-rados-wip-yuri3-testing-2022-02-28-0757-pacific-distro-default-smithi/6714778/orig.config.yaml
2022-03-02T03:06:25.739 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-03-01_17:45:51-rados-wip-yuri3-testing-2022-02-28-0757-pacific-distro-default-smithi/6714778
2022-03-02T03:06:25.740 INFO:teuthology.dispatcher.supervisor:Job PID: 31856
2022-03-02T03:06:25.740 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-02T09:36:55.239 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-02T09:36:55.436 INFO:teuthology.kill:Killing Pids: {31856}
2022-03-02T09:36:56.626 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-03-02T09:36:56.627 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-03-02T09:36:56.628 INFO:teuthology.misc:Compressing logs...
2022-03-02T09:36:56.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2022-03-02T09:36:56.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com'
2022-03-02T09:36:56.872 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-03-02T09:36:57.048 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-03-02T09:36:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-03-02T09:36:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2022-03-02T09:36:57.130 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-03-02T09:36:57.131 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now
2022-03-02T09:36:57.132 INFO:teuthology.misc:Compressing logs...
2022-03-02T09:36:57.380 INFO:teuthology.misc:Compressing logs...
2022-03-02T09:36:57.602 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-mgr.smithi150.vjnvwc.log: file size changed while zipping
2022-03-02T09:37:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-03-02T09:37:13.596 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-03-02T09:37:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-03-02T09:37:25.100 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-mgr.smithi142.sweigl.log: file size changed while zipping
2022-03-02T09:37:28.555 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-mon.smithi142.log: file size changed while zipping
2022-03-02T09:37:32.085 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-03-02T09:37:45.779 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-03-02T09:37:45.806 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-03-02T09:37:48.060 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-mon.smithi150.log: file size changed while zipping
2022-03-02T09:38:00.791 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-03-02T09:38:04.134 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/dceebd72-99d6-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-03-02T09:38:30.255 INFO:teuthology.kill:No teuthology processes running
2022-03-02T09:38:31.372 INFO:teuthology.kill:Nuking machines: ['smithi142', 'smithi150']
2022-03-02T09:38:32.365 INFO:teuthology.kill:2022-03-02 09:38:32,364.364 INFO:teuthology.nuke:targets:
2022-03-02T09:38:32.366 INFO:teuthology.kill:  smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCqvaDZyyuqyBQGN7MIfns8B71UNnMx/AI/VV5vOjDH7asrairjCPZznTf6c9Tsddd0E8OPbif+UYBCUl7/CStJTTsXkOVIZkioW+1d1gUS3kfOVANfvd5f7tpkzW8sazxiiUICns+6ExW8QS5g01kXcFI61cdo9NzlMPo7gcnuXLIMB7Ae+w1+l7DkVtN8ZNf+5bdN0z2I21tAxiMEbBnzLpmfdm+h+tO5L22XoztpT1sK7jvsoE9ofK5JHFRqbBu1fWI/lhjscf1W3hCqbmm6HpPccLP6cV6HWB3oJpKjO/HfAeUTYqxQNexJm9ul5nALJTLdgWRQcTZvkZpBYrBFp9YcMgYe4ZRo34UPjEDSoBt2Dh6e4wGr+r7bXYJ1xaobVrDfsOi9rmWvEu4dA2qKjpY+UUNOxGR6uATOrLWe6wWeigpkwyw8wSANG6y611Z0CimqmpZg1a5ijK2EjSue3k5+E92H04t74bAyPbGNfXugnu95Frny2CaqdULxOM=
2022-03-02T09:38:32.366 INFO:teuthology.kill:  smithi150.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC79xO1H8lor3z/AuEv7ETiD3tv8xuKDATYCLj3IPdm0OjsuPQbORm6w/jz0d80pDBpKgQ9TU3CCqzeqMYNeuU3U4VRHDzNVQbRHycD3ug5WOOpO0x7L9Bp6CE3nXc3g0Gtn4LDcmlO2xVqZR7aQapWtfrYNSA3Nb2QevCmXxUjZLhyodZYFb4mifoAubYUBOrW3j6kJawAsL/BwdHrS+LwtXmxCvp0g0Txwq6cdcvEgr+fnBZyo3yfAjdBI1/xH40+JlGaz7k1lg1nU5kBZqg1ixfcfMKD/nZKHngaj4t1EnssGIVJUBvRC7Bxt9sZiM0+CB9gNa5Q17W9csfdNaMDxbLRkfOPSOFJEHT6QhT7Hl8g+GEurqPDIOFtbP1cvkAOTQxcgwzzYt/5R/PIblD+WhFiXEONYSLal6y46X78IpHBoTFo9mg+UARt8srrhfb/t3FNVTwcOPE1QBeNeNI/aQduAy3s07ikImmB2FH47AM4DzXHV1ZIgk/sHdonQYk=
2022-03-02T09:38:32.395 INFO:teuthology.kill:2022-03-02 09:38:32,394.394 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-02T09:38:32.399 INFO:teuthology.kill:2022-03-02 09:38:32,398.398 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-02T09:38:32.476 INFO:teuthology.kill:2022-03-02 09:38:32,475.475 INFO:teuthology.orchestra.console:Power off smithi142
2022-03-02T09:38:32.491 INFO:teuthology.kill:2022-03-02 09:38:32,490.490 INFO:teuthology.orchestra.console:Power off smithi150
2022-03-02T09:38:40.855 INFO:teuthology.kill:2022-03-02 09:38:40,855.855 INFO:teuthology.orchestra.console:Power off for smithi150 completed
2022-03-02T09:38:41.012 INFO:teuthology.kill:2022-03-02 09:38:41,012.012 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com
2022-03-02T09:38:44.960 INFO:teuthology.kill:2022-03-02 09:38:44,960.960 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2022-03-02T09:38:45.104 INFO:teuthology.kill:2022-03-02 09:38:45,104.104 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com
2022-03-02T09:40:45.432 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-02T09:40:45.533 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi142 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-01_04:17:02-fs-pacific-distro-default-smithi/6713351
2022-03-02T09:40:45.533 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi150 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-01_04:17:02-fs-pacific-distro-default-smithi/6713351