2022-03-13T04:52:19.112 INFO:root:teuthology version: 1.1.0-fbfc7b38
2022-03-13T04:52:19.233 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi111.front.sepia.ceph.com' reimaging
2022-03-13T04:52:19.234 INFO:teuthology.lock.ops:Updating [ubuntu@smithi111.front.sepia.ceph.com]: reset os type and version on server
2022-03-13T04:52:19.234 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server
2022-03-13T04:52:19.253 INFO:teuthology.lock.ops:Node 'ubuntu@smithi111.front.sepia.ceph.com' reimaging is complete
2022-03-13T04:52:19.253 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging
2022-03-13T04:52:19.254 INFO:teuthology.lock.ops:Updating [ubuntu@smithi162.front.sepia.ceph.com]: reset os type and version on server
2022-03-13T04:52:19.254 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server
2022-03-13T04:52:19.274 INFO:teuthology.lock.ops:Node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging is complete
2022-03-13T04:52:19.472 INFO:teuthology.provision.fog.smithi111:Scheduling deploy of centos 8.stream
2022-03-13T04:52:19.495 INFO:teuthology.provision.fog.smithi162:Scheduling deploy of centos 8.stream
2022-03-13T04:52:19.825 INFO:teuthology.orchestra.console:Power off smithi111
2022-03-13T04:52:19.863 INFO:teuthology.orchestra.console:Power off smithi162
2022-03-13T04:52:28.197 INFO:teuthology.orchestra.console:Power off for smithi111 completed
2022-03-13T04:52:28.298 INFO:teuthology.orchestra.console:Power on smithi111
2022-03-13T04:52:32.335 INFO:teuthology.orchestra.console:Power off for smithi162 completed
2022-03-13T04:52:32.437 INFO:teuthology.orchestra.console:Power on smithi162
2022-03-13T04:52:36.665 INFO:teuthology.orchestra.console:Power on for smithi111 completed
2022-03-13T04:52:36.687 INFO:teuthology.orchestra.console:Power on for smithi162 completed
2022-03-13T04:52:36.767 INFO:teuthology.provision.fog.smithi111:Waiting for deploy to finish
2022-03-13T04:52:36.788 INFO:teuthology.provision.fog.smithi162:Waiting for deploy to finish
2022-03-13T04:57:24.654 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-13T04:57:24.909 INFO:teuthology.provision.fog.smithi162:Node is ready
2022-03-13T04:57:24.928 INFO:teuthology.orchestra.run.smithi162.stdout:smithi162.front.sepia.ceph.com
2022-03-13T04:57:24.987 INFO:teuthology.orchestra.run.smithi162.stdout:172.21.15.162 smithi162.front.sepia.ceph.com smithi162
2022-03-13T04:57:25.331 INFO:teuthology.provision.fog.smithi162:Deploy complete!
2022-03-13T04:57:42.816 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-13T04:57:43.090 INFO:teuthology.provision.fog.smithi111:Node is ready
2022-03-13T04:57:43.110 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111.front.sepia.ceph.com
2022-03-13T04:57:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111 smithi111.front.sepia.ceph.com smithi111
2022-03-13T04:57:43.544 INFO:teuthology.provision.fog.smithi111:Deploy complete!
2022-03-13T04:57:43.697 INFO:teuthology.lock.ops:Checking smithi111.front.sepia.ceph.com
2022-03-13T04:57:43.697 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-13T04:57:43.717 INFO:teuthology.lock.ops:Checking smithi162.front.sepia.ceph.com
2022-03-13T04:57:43.718 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-13T04:57:43.734 INFO:teuthology.lock.ops:Updating [smithi111.front.sepia.ceph.com]: set os type and version on server
2022-03-13T04:57:43.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2022-03-13T04:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64
2022-03-13T04:57:43.986 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream"
2022-03-13T04:57:43.987 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8"
2022-03-13T04:57:43.987 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos"
2022-03-13T04:57:43.987 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora"
2022-03-13T04:57:43.988 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8"
2022-03-13T04:57:43.988 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8"
2022-03-13T04:57:43.988 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-13T04:57:43.988 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31"
2022-03-13T04:57:43.989 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-13T04:57:43.989 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/"
2022-03-13T04:57:43.989 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-13T04:57:43.989 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-13T04:57:43.990 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-13T04:57:43.991 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server
2022-03-13T04:57:44.009 INFO:teuthology.lock.ops:Updating [smithi162.front.sepia.ceph.com]: set os type and version on server
2022-03-13T04:57:44.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com'
2022-03-13T04:57:44.247 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64
2022-03-13T04:57:44.305 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream"
2022-03-13T04:57:44.305 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8"
2022-03-13T04:57:44.306 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos"
2022-03-13T04:57:44.306 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora"
2022-03-13T04:57:44.306 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8"
2022-03-13T04:57:44.307 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8"
2022-03-13T04:57:44.307 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-13T04:57:44.307 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31"
2022-03-13T04:57:44.308 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-13T04:57:44.308 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/"
2022-03-13T04:57:44.308 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-13T04:57:44.308 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-13T04:57:44.308 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-13T04:57:44.309 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server
2022-03-13T04:57:44.408 INFO:teuthology.dispatcher.supervisor:Running job 6734099
2022-03-13T04:57:44.408 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6734099 --name teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6734099/orig.config.yaml
2022-03-13T04:57:44.412 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6734099
2022-03-13T04:57:44.412 INFO:teuthology.dispatcher.supervisor:Job PID: 27648
2022-03-13T04:57:44.413 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-13T11:28:14.312 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-13T11:28:14.498 INFO:teuthology.kill:Killing Pids: {27648}
2022-03-13T11:28:15.728 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-03-13T11:28:15.729 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-03-13T11:28:15.729 INFO:teuthology.misc:Compressing logs...
2022-03-13T11:28:15.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2022-03-13T11:28:15.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com'
2022-03-13T11:28:15.954 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-03-13T11:28:16.147 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-03-13T11:28:16.223 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-03-13T11:28:16.223 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now
2022-03-13T11:28:16.261 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-03-13T11:28:16.262 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now
2022-03-13T11:28:16.263 INFO:teuthology.misc:Compressing logs...
2022-03-13T11:28:16.609 INFO:teuthology.misc:Compressing logs...
2022-03-13T11:28:32.939 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-03-13T11:28:33.320 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-03-13T11:28:34.612 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-mgr.smithi111.dfeyji.log: file size changed while zipping
2022-03-13T11:28:46.031 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-03-13T11:28:52.934 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-03-13T11:28:53.763 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-mgr.smithi162.phbrev.log: file size changed while zipping
2022-03-13T11:28:58.608 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-mon.smithi111.log: file size changed while zipping
2022-03-13T11:29:07.844 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-03-13T11:29:12.301 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-mon.smithi162.log: file size changed while zipping
2022-03-13T11:29:18.207 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-03-13T11:29:28.122 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-03-13T11:29:36.589 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/49a0122e-a28b-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-03-13T11:30:08.176 INFO:teuthology.kill:No teuthology processes running
2022-03-13T11:30:09.466 INFO:teuthology.kill:Nuking machines: ['smithi111', 'smithi162']
2022-03-13T11:30:10.506 INFO:teuthology.kill:2022-03-13 11:30:10,506.506 INFO:teuthology.nuke:targets:
2022-03-13T11:30:10.507 INFO:teuthology.kill:  smithi111.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkcTxcOfvpOsS1DnO72LKiTj4w548MTwxiFXceFZtwJu87APomatQAJDNVxOHrp6Q+Zx0UmhD20G1yWUzoN50e8wkoUzqbnmTF3Hm3cZH9TzVDGhAnirYcVLffL3ItHJTgysXjiSO8GMhxIiNRyi9N3XjnzptEWlsk9VNLEHEHI4RyK+ttFE+pz+/VMWjErdE+aWEOl32tXVW6wc0b0qh8jeHuWiFJga6innAJ+MYj+YMjO712dgUKPc3baUmb14CmSt9rJSZTz4g+ESSJa7JkaYh6LQnXTwnLr/K7AL+RHyTZMc6h2sKjTEMI26itnGSMwxRnOSA1iEK7r7/kKJ00C6GrKudLFOorA8ksJWDze3nljNx0HtVGkQiwnIjWlrRAtap6exOtURVZHNn4O2hCgheZ2KubFgz3oHpl9A+2rqWoQKtVyUcjjHflPthT8yd4Se5295u/lc1fG5IJnjoCfjeqzLmN4r23lH/8oAVdE+nw9FKl2zqrJ/SrOgUw5rk=
2022-03-13T11:30:10.507 INFO:teuthology.kill:  smithi162.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN+E1jgF1uHrmsSV45PJdaiFoZKVmebA2z0uF/qiqRRSwafD3JQDy060xv0BGFlJzpVOceDCQ6vBXpEHyYTP2/TavsGeUhLfkIWl8iJyUaMFVBo+/hVaY6j9mils6JOpGBXaNzDCGkFBmsqGQtegfa/+1vNzrZASwxSBddIBjbQixGt8b8wkQOAmFHrUlp0hcTYwaI9Jj0D89+lA+9qcYgBV2cKDkrrcQ+3YgKWue/tErB8s2gR8GUFD4i4Np89xBCY8enZVz79/1O1DxyBg1qjA20+uP0+jFD4OmGVMpY+PdQoUN2HE8BSBuKuXzNKSzLmKUEw5FJhAnHBsflfsIIm2bMNcesS/xbOPv5umfTA3rzmS9i6MXmG91NqhaDdYdNJ4bEoNs7CWQw/99ELl8PO5KqDgjXfPra12RuWKiAeybHq58GcAWY2i3uFX0zEWfgxoidIkovqb19DJqYmQSDtoZkWuROcIgIY+knjv8oWxTnD67DyZGawXVkbFf+n38=
2022-03-13T11:30:10.537 INFO:teuthology.kill:2022-03-13 11:30:10,537.537 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-13T11:30:10.541 INFO:teuthology.kill:2022-03-13 11:30:10,541.541 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-13T11:30:10.616 INFO:teuthology.kill:2022-03-13 11:30:10,616.616 INFO:teuthology.orchestra.console:Power off smithi162
2022-03-13T11:30:10.630 INFO:teuthology.kill:2022-03-13 11:30:10,630.630 INFO:teuthology.orchestra.console:Power off smithi111
2022-03-13T11:30:18.984 INFO:teuthology.kill:2022-03-13 11:30:18,984.984 INFO:teuthology.orchestra.console:Power off for smithi162 completed
2022-03-13T11:30:18.995 INFO:teuthology.kill:2022-03-13 11:30:18,995.995 INFO:teuthology.orchestra.console:Power off for smithi111 completed
2022-03-13T11:30:19.130 INFO:teuthology.kill:2022-03-13 11:30:19,130.130 INFO:teuthology.lock.ops:unlocked: smithi162.front.sepia.ceph.com
2022-03-13T11:30:19.138 INFO:teuthology.kill:2022-03-13 11:30:19,138.138 INFO:teuthology.lock.ops:unlocked: smithi111.front.sepia.ceph.com
2022-03-13T11:32:19.458 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-13T11:32:19.564 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi111 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-13_07:01:03-rados-master-distro-default-smithi/6734801
2022-03-13T11:32:19.564 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi162 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-13_07:01:03-rados-master-distro-default-smithi/6734801