2021-12-19T22:47:58.528 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-19T22:47:58.668 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi001.front.sepia.ceph.com' reimaging 2021-12-19T22:47:58.669 INFO:teuthology.lock.ops:Updating [ubuntu@smithi001.front.sepia.ceph.com]: reset os type and version on server 2021-12-19T22:47:58.670 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2021-12-19T22:47:58.691 INFO:teuthology.lock.ops:Node 'ubuntu@smithi001.front.sepia.ceph.com' reimaging is complete 2021-12-19T22:47:58.691 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi068.front.sepia.ceph.com' reimaging 2021-12-19T22:47:58.691 INFO:teuthology.lock.ops:Updating [ubuntu@smithi068.front.sepia.ceph.com]: reset os type and version on server 2021-12-19T22:47:58.692 INFO:teuthology.lock.ops:Updating smithi068.front.sepia.ceph.com on lock server 2021-12-19T22:47:58.714 INFO:teuthology.lock.ops:Node 'ubuntu@smithi068.front.sepia.ceph.com' reimaging is complete 2021-12-19T22:47:58.908 INFO:teuthology.provision.fog.smithi001:Scheduling deploy of centos 8.stream 2021-12-19T22:47:58.926 INFO:teuthology.provision.fog.smithi068:Scheduling deploy of centos 8.stream 2021-12-19T22:47:59.250 INFO:teuthology.orchestra.console:Power off smithi001 2021-12-19T22:47:59.265 INFO:teuthology.orchestra.console:Power off smithi068 2021-12-19T22:48:07.623 INFO:teuthology.orchestra.console:Power off for smithi001 completed 2021-12-19T22:48:07.630 INFO:teuthology.orchestra.console:Power off for smithi068 completed 2021-12-19T22:48:07.725 INFO:teuthology.orchestra.console:Power on smithi001 2021-12-19T22:48:07.733 INFO:teuthology.orchestra.console:Power on smithi068 2021-12-19T22:48:16.098 INFO:teuthology.orchestra.console:Power on for smithi001 completed 2021-12-19T22:48:16.107 INFO:teuthology.orchestra.console:Power on for smithi068 completed 2021-12-19T22:48:16.201 INFO:teuthology.provision.fog.smithi001:Waiting for deploy to finish 2021-12-19T22:48:16.208 INFO:teuthology.provision.fog.smithi068:Waiting for deploy to finish 2021-12-19T22:51:57.088 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-19T22:52:12.175 INFO:teuthology.provision.fog.smithi001:Node is ready 2021-12-19T22:52:12.196 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001.front.sepia.ceph.com 2021-12-19T22:52:12.256 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1 smithi001.front.sepia.ceph.com smithi001 2021-12-19T22:52:12.611 INFO:teuthology.provision.fog.smithi001:Deploy complete! 2021-12-19T22:53:21.558 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-19T22:53:21.827 INFO:teuthology.provision.fog.smithi068:Node is ready 2021-12-19T22:53:21.844 INFO:teuthology.orchestra.run.smithi068.stdout:smithi068.front.sepia.ceph.com 2021-12-19T22:53:21.901 INFO:teuthology.orchestra.run.smithi068.stdout:172.21.15.68 smithi068.front.sepia.ceph.com smithi068 2021-12-19T22:53:22.232 INFO:teuthology.provision.fog.smithi068:Deploy complete! 2021-12-19T22:53:22.382 INFO:teuthology.lock.ops:Checking smithi001.front.sepia.ceph.com 2021-12-19T22:53:22.383 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-19T22:53:22.410 INFO:teuthology.lock.ops:Checking smithi068.front.sepia.ceph.com 2021-12-19T22:53:22.411 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-19T22:53:22.429 INFO:teuthology.lock.ops:Updating [smithi001.front.sepia.ceph.com]: set os type and version on server 2021-12-19T22:53:22.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2021-12-19T22:53:22.680 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2021-12-19T22:53:22.738 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2021-12-19T22:53:22.738 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="8" 2021-12-19T22:53:22.738 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2021-12-19T22:53:22.738 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2021-12-19T22:53:22.739 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="8" 2021-12-19T22:53:22.739 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el8" 2021-12-19T22:53:22.739 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-19T22:53:22.739 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2021-12-19T22:53:22.740 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-19T22:53:22.740 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2021-12-19T22:53:22.740 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-19T22:53:22.740 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-19T22:53:22.741 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-19T22:53:22.742 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2021-12-19T22:53:22.761 INFO:teuthology.lock.ops:Updating [smithi068.front.sepia.ceph.com]: set os type and version on server 2021-12-19T22:53:22.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi068.front.sepia.ceph.com' 2021-12-19T22:53:22.952 INFO:teuthology.orchestra.run.smithi068.stdout:x86_64 2021-12-19T22:53:23.009 INFO:teuthology.orchestra.run.smithi068.stdout:NAME="CentOS Stream" 2021-12-19T22:53:23.009 INFO:teuthology.orchestra.run.smithi068.stdout:VERSION="8" 2021-12-19T22:53:23.009 INFO:teuthology.orchestra.run.smithi068.stdout:ID="centos" 2021-12-19T22:53:23.010 INFO:teuthology.orchestra.run.smithi068.stdout:ID_LIKE="rhel fedora" 2021-12-19T22:53:23.010 INFO:teuthology.orchestra.run.smithi068.stdout:VERSION_ID="8" 2021-12-19T22:53:23.010 INFO:teuthology.orchestra.run.smithi068.stdout:PLATFORM_ID="platform:el8" 2021-12-19T22:53:23.010 INFO:teuthology.orchestra.run.smithi068.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-19T22:53:23.010 INFO:teuthology.orchestra.run.smithi068.stdout:ANSI_COLOR="0;31" 2021-12-19T22:53:23.011 INFO:teuthology.orchestra.run.smithi068.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-19T22:53:23.011 INFO:teuthology.orchestra.run.smithi068.stdout:HOME_URL="https://centos.org/" 2021-12-19T22:53:23.011 INFO:teuthology.orchestra.run.smithi068.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-19T22:53:23.011 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-19T22:53:23.012 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-19T22:53:23.012 INFO:teuthology.lock.ops:Updating smithi068.front.sepia.ceph.com on lock server 2021-12-19T22:53:23.106 INFO:teuthology.dispatcher.supervisor:Running job 6572152 2021-12-19T22:53:23.106 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-12-19_07:01:03-rados-master-distro-default-smithi/6572152 --name teuthology-2021-12-19_07:01:03-rados-master-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/teuthology-2021-12-19_07:01:03-rados-master-distro-default-smithi/6572152/orig.config.yaml 2021-12-19T22:53:23.110 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-12-19_07:01:03-rados-master-distro-default-smithi/6572152 2021-12-19T22:53:23.111 INFO:teuthology.dispatcher.supervisor:Job PID: 11966 2021-12-19T22:53:23.111 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-12-20T10:53:43.752 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-12-20T10:53:43.926 INFO:teuthology.kill:Killing Pids: {11966} 2021-12-20T10:53:45.060 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-12-20T10:53:45.061 INFO:teuthology.task.internal:roles: ubuntu@smithi068.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-12-20T10:53:45.061 INFO:teuthology.misc:Compressing logs... 2021-12-20T10:53:45.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2021-12-20T10:53:45.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi068.front.sepia.ceph.com' 2021-12-20T10:53:45.300 INFO:teuthology.orchestra.run.smithi001.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-12-20T10:53:45.486 INFO:teuthology.orchestra.run.smithi068.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-12-20T10:53:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-12-20T10:53:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:tar: Error is not recoverable: exiting now 2021-12-20T10:53:45.566 INFO:teuthology.orchestra.run.smithi068.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-12-20T10:53:45.566 INFO:teuthology.orchestra.run.smithi068.stderr:tar: Error is not recoverable: exiting now 2021-12-20T10:53:45.568 INFO:teuthology.misc:Compressing logs... 2021-12-20T10:53:45.971 INFO:teuthology.orchestra.run.smithi001.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-20T10:53:46.177 INFO:teuthology.misc:Compressing logs... 2021-12-20T10:53:53.295 INFO:teuthology.orchestra.run.smithi068.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-mon.smithi068.log: file size changed while zipping 2021-12-20T10:53:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-12-20T10:54:07.737 INFO:teuthology.orchestra.run.smithi068.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2021-12-20T10:54:12.743 INFO:teuthology.orchestra.run.smithi001.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-mon.smithi001.log: file size changed while zipping 2021-12-20T10:54:20.183 INFO:teuthology.orchestra.run.smithi068.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-12-20T10:54:23.148 INFO:teuthology.orchestra.run.smithi068.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-mgr.smithi068.aljyuo.log: file size changed while zipping 2021-12-20T10:54:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-12-20T10:54:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-12-20T10:54:37.468 INFO:teuthology.orchestra.run.smithi068.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-12-20T10:54:48.061 INFO:teuthology.orchestra.run.smithi001.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-12-20T10:54:49.204 INFO:teuthology.orchestra.run.smithi068.stderr:gzip: /var/log/ceph/3cf72434-611f-11ec-8c31-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2021-12-20T10:55:38.556 INFO:teuthology.kill:No teuthology processes running 2021-12-20T10:55:39.613 INFO:teuthology.kill:Nuking machines: ['smithi001', 'smithi068'] 2021-12-20T10:55:40.490 INFO:teuthology.kill:2021-12-20 10:55:40,489.489 INFO:teuthology.nuke:targets: 2021-12-20T10:55:40.491 INFO:teuthology.kill: smithi001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHXH+zfEFcAQTjx+nn4Eoe0pJGtixumA7BauQmSv4KW6d8sOCSDTkBErNMhSTRvsEwktPESKUt8SFOFfdD5qhPKBy9LMGDLwnAmG1ZqeuVwzESrqJn1yIsrBJJnsApPRQG3r0RRFrhjGe4PCSrlOkosg0JzyQCZeplJKlz4ThJmBFSBoJ+AzmI/Fia/SJhwGIj0gMmXmCg60SR1alVxtmvSBaXn/CjQsu/G+e74JqWQU2pmKRt5Ueq7Gw3vkwk8MXI98Z25deN1EfSWG6eQlxir0PRMjJ7zjnygjz+yosjoVfjv9hvnCEoVG8dIyBHgKouXZE/ZUQp1AJtoXZhjza41mH8+UyzXMctJN75PyniDwwURXDIqh+hcOi8WBq5GMLeDl4alAuMcBg8E432v5sNLl5ZGt8ac7CvToPwZeW/ZdstguLJmifSpKs7UzMsnScKsW+IEvWFmLxj5YKyd4uvMN5lNfabBuDakHgCtw0KieUXE62/8pFSkwI5x6i2Eds= 2021-12-20T10:55:40.491 INFO:teuthology.kill: smithi068.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMx/y6r6VdeRna6fpkiMGw3b5k8zmVRXHCF5yqST/S6x1xgpuJWKOAoyPZu25kHwSemS7zaYpiSBb4VrgKbWmpO1PobiElzUkXTMCr92MrLdgbzGAKng8PnnBNS7Zt41OcCkKmeBL2GPWk7F5N9LrHQW8ivhmHGghS31sUROaEdYsVszcfbGi+tti2A5/BWb0hQ4tKk1+UYPgsKjj3czseRhzLkHxVX5mY63LZEPvEUVK40bQmyriib1gMCo2ABne7O8L0Y19TRS5zJCKUwS3ZPkICy/PVADQAb+dPai1e/rDkxw5jnrMO0eAdZXDR8TS3gc4EZRfGc4PWO8WlgDP/DkVxRzcbF72+OLY7UX4o1iXAcoUXhtizYi/0vZpNgskCPJ7/lsCL2iGyhAm9YuOlQeHVkI9LsRp/3eCXrlXbMOTJcrOGEG7VaflnTQ9SsMHywVNvbbCOyBH7WP/c00f2g8Yh95wxmSdAFWvqxFlJFDJaEoeBRdA3g55jpZlj78k= 2021-12-20T10:55:40.516 INFO:teuthology.kill:2021-12-20 10:55:40,516.516 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-20T10:55:40.519 INFO:teuthology.kill:2021-12-20 10:55:40,519.519 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-20T10:55:40.594 INFO:teuthology.kill:2021-12-20 10:55:40,594.594 INFO:teuthology.orchestra.console:Power off smithi068 2021-12-20T10:55:40.606 INFO:teuthology.kill:2021-12-20 10:55:40,606.606 INFO:teuthology.orchestra.console:Power off smithi001 2021-12-20T10:55:53.078 INFO:teuthology.kill:2021-12-20 10:55:53,077.077 INFO:teuthology.orchestra.console:Power off for smithi068 completed 2021-12-20T10:55:53.083 INFO:teuthology.kill:2021-12-20 10:55:53,083.083 INFO:teuthology.orchestra.console:Power off for smithi001 completed 2021-12-20T10:55:53.233 INFO:teuthology.kill:2021-12-20 10:55:53,232.232 INFO:teuthology.lock.ops:unlocked: smithi068.front.sepia.ceph.com 2021-12-20T10:55:53.236 INFO:teuthology.kill:2021-12-20 10:55:53,236.236 INFO:teuthology.lock.ops:unlocked: smithi001.front.sepia.ceph.com 2021-12-20T10:57:53.449 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15