2022-03-28T22:27:50.567 INFO:root:teuthology version: 1.1.0-70390753
2022-03-28T22:27:50.703 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi136.front.sepia.ceph.com' reimaging
2022-03-28T22:27:50.704 INFO:teuthology.lock.ops:Updating [ubuntu@smithi136.front.sepia.ceph.com]: reset os type and version on server
2022-03-28T22:27:50.704 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server
2022-03-28T22:27:50.723 INFO:teuthology.lock.ops:Node 'ubuntu@smithi136.front.sepia.ceph.com' reimaging is complete
2022-03-28T22:27:50.724 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi152.front.sepia.ceph.com' reimaging
2022-03-28T22:27:50.724 INFO:teuthology.lock.ops:Updating [ubuntu@smithi152.front.sepia.ceph.com]: reset os type and version on server
2022-03-28T22:27:50.724 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server
2022-03-28T22:27:50.745 INFO:teuthology.lock.ops:Node 'ubuntu@smithi152.front.sepia.ceph.com' reimaging is complete
2022-03-28T22:27:50.943 INFO:teuthology.provision.fog.smithi136:Scheduling deploy of centos 8.stream
2022-03-28T22:27:50.951 INFO:teuthology.provision.fog.smithi152:Scheduling deploy of centos 8.stream
2022-03-28T22:27:51.266 INFO:teuthology.orchestra.console:Power off smithi136
2022-03-28T22:27:51.281 INFO:teuthology.orchestra.console:Power off smithi152
2022-03-28T22:27:51.414 INFO:teuthology.orchestra.console:Power off for smithi136 completed
2022-03-28T22:27:51.427 INFO:teuthology.orchestra.console:Power off for smithi152 completed
2022-03-28T22:27:51.517 INFO:teuthology.orchestra.console:Power on smithi136
2022-03-28T22:27:51.528 INFO:teuthology.orchestra.console:Power on smithi152
2022-03-28T22:27:55.776 INFO:teuthology.orchestra.console:Power on for smithi136 completed
2022-03-28T22:27:55.877 INFO:teuthology.provision.fog.smithi136:Waiting for deploy to finish
2022-03-28T22:27:59.895 INFO:teuthology.orchestra.console:Power on for smithi152 completed
2022-03-28T22:27:59.997 INFO:teuthology.provision.fog.smithi152:Waiting for deploy to finish
2022-03-28T22:30:31.359 ERROR:teuthology.orchestra.connection:Error authenticating with smithi152.front.sepia.ceph.com: Authentication failed.
2022-03-28T22:32:42.943 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-28T22:32:43.191 INFO:teuthology.provision.fog.smithi136:Node is ready
2022-03-28T22:32:43.249 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136.front.sepia.ceph.com
2022-03-28T22:32:43.307 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136 smithi136.front.sepia.ceph.com smithi136
2022-03-28T22:32:43.660 INFO:teuthology.provision.fog.smithi136:Deploy complete!
2022-03-28T22:32:44.691 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-28T22:32:44.981 INFO:teuthology.provision.fog.smithi152:Node is ready
2022-03-28T22:32:44.999 INFO:teuthology.orchestra.run.smithi152.stdout:smithi152.front.sepia.ceph.com
2022-03-28T22:32:45.056 INFO:teuthology.orchestra.run.smithi152.stdout:172.21.15.152 smithi152.front.sepia.ceph.com smithi152
2022-03-28T22:32:45.405 INFO:teuthology.provision.fog.smithi152:Deploy complete!
2022-03-28T22:32:45.563 INFO:teuthology.lock.ops:Checking smithi136.front.sepia.ceph.com
2022-03-28T22:32:45.564 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-28T22:32:45.588 INFO:teuthology.lock.ops:Checking smithi152.front.sepia.ceph.com
2022-03-28T22:32:45.588 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-28T22:32:45.613 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: set os type and version on server
2022-03-28T22:32:45.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com'
2022-03-28T22:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64
2022-03-28T22:32:45.868 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream"
2022-03-28T22:32:45.868 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8"
2022-03-28T22:32:45.868 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos"
2022-03-28T22:32:45.869 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora"
2022-03-28T22:32:45.869 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8"
2022-03-28T22:32:45.869 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8"
2022-03-28T22:32:45.869 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-28T22:32:45.870 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31"
2022-03-28T22:32:45.870 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-28T22:32:45.870 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/"
2022-03-28T22:32:45.870 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-28T22:32:45.870 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-28T22:32:45.871 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-28T22:32:45.872 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server
2022-03-28T22:32:45.891 INFO:teuthology.lock.ops:Updating [smithi152.front.sepia.ceph.com]: set os type and version on server
2022-03-28T22:32:45.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com'
2022-03-28T22:32:46.081 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64
2022-03-28T22:32:46.138 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream"
2022-03-28T22:32:46.139 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="8"
2022-03-28T22:32:46.139 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos"
2022-03-28T22:32:46.139 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora"
2022-03-28T22:32:46.139 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="8"
2022-03-28T22:32:46.140 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el8"
2022-03-28T22:32:46.140 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-28T22:32:46.140 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31"
2022-03-28T22:32:46.141 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-28T22:32:46.141 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/"
2022-03-28T22:32:46.141 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-28T22:32:46.142 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-28T22:32:46.142 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-28T22:32:46.143 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server
2022-03-28T22:32:46.263 INFO:teuthology.dispatcher.supervisor:Running job 6762904
2022-03-28T22:32:46.263 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_70390753284281fb41f7552547fa9eb007fa4eef/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-03-27_03:31:03-rados-pacific-distro-default-smithi/6762904 --name teuthology-2022-03-27_03:31:03-rados-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/teuthology-2022-03-27_03:31:03-rados-pacific-distro-default-smithi/6762904/orig.config.yaml
2022-03-28T22:32:46.267 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-03-27_03:31:03-rados-pacific-distro-default-smithi/6762904
2022-03-28T22:32:46.268 INFO:teuthology.dispatcher.supervisor:Job PID: 8603
2022-03-28T22:32:46.268 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-29T05:02:57.052 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-29T05:02:57.286 INFO:teuthology.kill:Killing Pids: {8603}
2022-03-29T05:02:58.421 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-03-29T05:02:58.422 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-03-29T05:02:58.423 INFO:teuthology.misc:Compressing logs...
2022-03-29T05:02:58.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com'
2022-03-29T05:02:58.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com'
2022-03-29T05:02:58.646 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-03-29T05:02:58.852 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-03-29T05:02:58.900 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-03-29T05:02:58.900 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now
2022-03-29T05:02:58.978 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-03-29T05:02:58.979 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now
2022-03-29T05:02:58.980 INFO:teuthology.misc:Compressing logs...
2022-03-29T05:02:59.199 INFO:teuthology.misc:Compressing logs...
2022-03-29T05:03:00.032 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-mon.smithi152.log: file size changed while zipping
2022-03-29T05:03:01.954 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-03-29T05:03:02.012 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-03-29T05:03:04.109 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-03-29T05:03:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-03-29T05:03:05.904 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-03-29T05:03:07.855 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-03-29T05:03:08.085 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-03-29T05:03:11.245 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-mon.smithi136.log: file size changed while zipping
2022-03-29T05:03:11.935 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-mgr.smithi136.fuknew.log: file size changed while zipping
2022-03-29T05:03:15.335 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/2e34c19c-aee8-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-03-29T05:03:52.436 INFO:teuthology.kill:No teuthology processes running
2022-03-29T05:03:53.782 INFO:teuthology.kill:Nuking machines: ['smithi136', 'smithi152']
2022-03-29T05:03:54.720 INFO:teuthology.kill:2022-03-29 05:03:54,720.720 INFO:teuthology.nuke:targets:
2022-03-29T05:03:54.721 INFO:teuthology.kill:  smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChMukTwY5z7EdsDFnm2ITp/RrE1/JSMYY0FtC5tRXhmm3EVbwNzHtmJB2KAmSl70Mzf9RzlVNV2v0PpQRi2vS019UEgo0v/yG1nQWjmDmoll1svmKWa70G4cLVtEeTTR+tDvUIaj9nAti/XtS2zi08yEcfY6aRK005z7CKMN+SvQBGqAOAELQ7IgBBFKowWHWR9qpxumG883tpihHVerkjAgN0NgI7146GB/27B7szWshZrGoKf1E2f/em7EUljyw4VvmTM8iF4WpS/l6z5Vc6QheuNUcLrmm8I//KB1BFOpPY4mbr/4CXTuqRvhZc9GXdIHRP/P9VKeXmdvWE6fy3O0lhd5LOZwosLp8p/EkJQnvcp8EUueMun2NoGu4X98hF2IvhBUpgxWVfzEmK1HKDWTrxfVMJVfVqxwMK//+ayoU6v7WlhYZYsZmnGel+mHpKcK3VPSwrLf8NuClslSTXiqx+ldIw1irvnZVVnqLAexVhrbI+I/kHXuowJgLfKcc=
2022-03-29T05:03:54.722 INFO:teuthology.kill:  smithi152.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNnsUPB2Sr1cH8XFJY5Cwv6Qr702H0fmU62zpv5VVDMh+7qD90kXcXTMzdMBSks8nlP8yPIZyL0BT3COZg1Tq6gMHh1mSI0U1jmbXbIBWAIBdxTNSe7BbaieaGzodrsTKWSI8eALPx/TWu9mRwE7qsKMWxmSe7nR6D7DrdufU4g4SIxOTZ+flk/7eYl6A2v/XZMJVnqf+4tIALVt5z91SHw0Gs8pGG0+heUD8DhZx/ogJGdJuig8pBug95r4I8GMPN1zhYfLnGdgJyC5PwRee/spFfA0rESO8frx/yCquqn3kKqLl6KJSnTQZgWen/zwLxQTuSM3aRDiFAbDDol/MLEDi+NRBWJObH9OkHavunyZsU2JgP/JzZceCT2ZBZ2UED/NeKjaKX0+SKOl3n3jijvYkYxgOp3QH34i230zFRKKhjV8KqafnmHHm1S6yJg4osrMFe5ROIuM9g406TlweaU6bVz1VFyvXfpL84ZAz3AvpnvC2pbe+Kz7I5ShTNNls=
2022-03-29T05:03:54.748 INFO:teuthology.kill:2022-03-29 05:03:54,748.748 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-29T05:03:54.752 INFO:teuthology.kill:2022-03-29 05:03:54,752.752 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-29T05:03:54.825 INFO:teuthology.kill:2022-03-29 05:03:54,825.825 INFO:teuthology.orchestra.console:Power off smithi136
2022-03-29T05:03:54.840 INFO:teuthology.kill:2022-03-29 05:03:54,839.839 INFO:teuthology.orchestra.console:Power off smithi152
2022-03-29T05:04:03.201 INFO:teuthology.kill:2022-03-29 05:04:03,201.201 INFO:teuthology.orchestra.console:Power off for smithi152 completed
2022-03-29T05:04:03.347 INFO:teuthology.kill:2022-03-29 05:04:03,346.346 INFO:teuthology.lock.ops:unlocked: smithi152.front.sepia.ceph.com
2022-03-29T05:04:07.305 INFO:teuthology.kill:2022-03-29 05:04:07,305.305 INFO:teuthology.orchestra.console:Power off for smithi136 completed
2022-03-29T05:04:07.466 INFO:teuthology.kill:2022-03-29 05:04:07,466.466 INFO:teuthology.lock.ops:unlocked: smithi136.front.sepia.ceph.com
2022-03-29T05:06:07.712 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-29T05:06:07.808 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi152 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-29_05:00:02-smoke-master-distro-default-smithi/6766219