2023-01-12T13:16:33.364 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-12T13:16:45.282 INFO:teuthology.lock.ops:Start node 'smithi073.front.sepia.ceph.com' reimaging 2023-01-12T13:16:45.283 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: reset os type and version on server 2023-01-12T13:16:45.284 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2023-01-12T13:16:45.303 INFO:teuthology.lock.ops:Node 'smithi073.front.sepia.ceph.com' reimaging is complete 2023-01-12T13:16:45.303 INFO:teuthology.lock.ops:Start node 'smithi084.front.sepia.ceph.com' reimaging 2023-01-12T13:16:45.304 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: reset os type and version on server 2023-01-12T13:16:45.304 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2023-01-12T13:16:45.320 INFO:teuthology.lock.ops:Node 'smithi084.front.sepia.ceph.com' reimaging is complete 2023-01-12T13:16:45.456 INFO:teuthology.provision.fog.smithi073:Scheduling deploy of centos 8.stream 2023-01-12T13:16:45.498 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of centos 8.stream 2023-01-12T13:16:45.740 INFO:teuthology.orchestra.console:Power off smithi073 2023-01-12T13:16:45.783 INFO:teuthology.orchestra.console:Power off smithi084 2023-01-12T13:16:45.893 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2023-01-12T13:16:45.994 INFO:teuthology.orchestra.console:Power on smithi073 2023-01-12T13:17:00.605 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2023-01-12T13:17:00.706 INFO:teuthology.orchestra.console:Power on smithi084 2023-01-12T13:17:04.961 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2023-01-12T13:17:05.062 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2023-01-12T13:17:06.849 INFO:teuthology.orchestra.console:Power on for smithi073 completed 2023-01-12T13:17:06.950 INFO:teuthology.provision.fog.smithi073:Waiting for deploy to finish 2023-01-12T13:19:48.617 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73 2023-01-12T13:19:57.708 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73 2023-01-12T13:20:07.885 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-01-12T13:20:10.956 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73 2023-01-12T13:20:22.109 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-01-12T13:20:25.164 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73 2023-01-12T13:20:31.184 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-01-12T13:21:02.966 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-12T13:21:23.685 INFO:teuthology.provision.fog.smithi073:Node is ready 2023-01-12T13:21:23.705 INFO:teuthology.orchestra.run.smithi073.stdout:smithi001 2023-01-12T13:21:23.762 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.1 smithi001 smithi001 2023-01-12T13:21:24.127 INFO:teuthology.provision.fog.smithi073:Deploy complete! 2023-01-12T13:21:38.615 WARNING:teuthology.provision.fog:timed out 2023-01-12T13:21:44.916 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-12T13:21:45.174 INFO:teuthology.provision.fog.smithi084:Node is ready 2023-01-12T13:21:45.234 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2023-01-12T13:21:45.293 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2023-01-12T13:21:45.648 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2023-01-12T13:21:45.858 INFO:teuthology.lock.ops:Checking smithi073.front.sepia.ceph.com 2023-01-12T13:21:45.859 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-12T13:21:45.877 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2023-01-12T13:21:45.877 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-12T13:21:45.892 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: set os type and version on server 2023-01-12T13:21:45.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-01-12T13:21:46.142 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2023-01-12T13:21:46.199 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream" 2023-01-12T13:21:46.199 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="8" 2023-01-12T13:21:46.199 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2023-01-12T13:21:46.200 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2023-01-12T13:21:46.200 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="8" 2023-01-12T13:21:46.200 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el8" 2023-01-12T13:21:46.200 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-12T13:21:46.201 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2023-01-12T13:21:46.201 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-12T13:21:46.201 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/" 2023-01-12T13:21:46.201 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-12T13:21:46.201 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-12T13:21:46.202 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-12T13:21:46.203 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2023-01-12T13:21:51.227 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2023-01-12T13:21:51.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-01-12T13:21:51.431 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2023-01-12T13:21:51.490 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2023-01-12T13:21:51.490 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8" 2023-01-12T13:21:51.490 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2023-01-12T13:21:51.490 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2023-01-12T13:21:51.491 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8" 2023-01-12T13:21:51.491 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2023-01-12T13:21:51.491 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-12T13:21:51.491 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2023-01-12T13:21:51.491 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-12T13:21:51.492 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2023-01-12T13:21:51.492 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-12T13:21:51.492 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-12T13:21:51.492 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-12T13:21:51.493 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2023-01-12T13:21:58.545 INFO:teuthology.dispatcher.supervisor:Running job 7135295 2023-01-12T13:21:58.546 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2023-01-12_12:11:23-orch:cephadm-wip-adk-testing-2023-01-11-2322-distro-default-smithi/7135295 --name adking-2023-01-12_12:11:23-orch:cephadm-wip-adk-testing-2023-01-11-2322-distro-default-smithi --description orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} -- /home/teuthworker/archive/adking-2023-01-12_12:11:23-orch:cephadm-wip-adk-testing-2023-01-11-2322-distro-default-smithi/7135295/orig.config.yaml 2023-01-12T13:21:58.550 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/adking-2023-01-12_12:11:23-orch:cephadm-wip-adk-testing-2023-01-11-2322-distro-default-smithi/7135295 2023-01-12T13:21:58.551 INFO:teuthology.dispatcher.supervisor:Job PID: 32607 2023-01-12T13:21:58.551 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-01-13T01:23:04.918 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-13T01:23:05.092 INFO:teuthology.kill:Killing Pids: {32607} 2023-01-13T01:23:05.952 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['host.a', 'client.0'] 2023-01-13T01:23:05.953 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['host.b', 'client.1'] 2023-01-13T01:23:05.953 INFO:teuthology.misc:Compressing logs... 2023-01-13T01:23:05.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-01-13T01:23:06.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-01-13T01:23:06.228 INFO:teuthology.orchestra.run.smithi073.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-13T01:23:06.420 INFO:teuthology.orchestra.run.smithi084.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-13T01:23:06.534 INFO:teuthology.orchestra.run.smithi073.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-13T01:23:06.535 INFO:teuthology.orchestra.run.smithi073.stderr:tar: Error is not recoverable: exiting now 2023-01-13T01:23:06.714 INFO:teuthology.orchestra.run.smithi084.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-13T01:23:06.714 INFO:teuthology.orchestra.run.smithi084.stderr:tar: Error is not recoverable: exiting now 2023-01-13T01:23:06.716 INFO:teuthology.misc:Compressing logs... 2023-01-13T01:23:06.830 INFO:teuthology.orchestra.run.smithi084.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2023-01-13T01:23:06.951 INFO:teuthology.misc:Compressing logs... 2023-01-13T01:23:20.522 INFO:teuthology.orchestra.run.smithi084.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2023-01-13T01:23:20.792 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2023-01-13T01:23:27.830 INFO:teuthology.orchestra.run.smithi084.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-mon.smithi084.log: file size changed while zipping 2023-01-13T01:23:34.493 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2023-01-13T01:23:45.376 INFO:teuthology.orchestra.run.smithi084.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2023-01-13T01:23:49.647 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2023-01-13T01:24:02.577 INFO:teuthology.orchestra.run.smithi084.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2023-01-13T01:24:08.639 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-mon.smithi073.log: file size changed while zipping 2023-01-13T01:24:16.826 INFO:teuthology.orchestra.run.smithi084.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2023-01-13T01:24:26.934 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2023-01-13T01:24:30.986 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/64f2a4b4-927d-11ed-821d-001a4aab830c/ceph-mgr.smithi073.jwmxeu.log: file size changed while zipping 2023-01-13T01:25:03.794 INFO:teuthology.kill:No teuthology processes running 2023-01-13T01:25:04.540 INFO:teuthology.kill:Nuking machines: ['smithi073', 'smithi084'] 2023-01-13T01:25:05.169 INFO:teuthology.kill:2023-01-13 01:25:05,169.169 INFO:teuthology.nuke:targets: 2023-01-13T01:25:05.170 INFO:teuthology.kill: smithi073.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBLC9Z7WiOZ9ZFEY+gdMlI68j5SgFb5WEcq1yrjXB+wQMfqhK5OAvzpJ6f5lM6Du06HKikWAQ7PFIsK/wvA7k8w= 2023-01-13T01:25:05.170 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKsCKBwTetzMkhS9WpLpLMaVIVqasgrWrWTZzrEP0zLN+q60CFEc+r9o6S8uLktvZYuzruZ2+kiTkERrRKg4GBw= 2023-01-13T01:25:05.170 INFO:teuthology.kill:2023-01-13 01:25:05,169.169 INFO:teuthology.nuke:Checking targets against current locks 2023-01-13T01:25:05.219 INFO:teuthology.kill:2023-01-13 01:25:05,219.219 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-13T01:25:05.237 INFO:teuthology.kill:2023-01-13 01:25:05,237.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-13T01:25:05.294 INFO:teuthology.kill:2023-01-13 01:25:05,294.294 INFO:teuthology.orchestra.console:Power off smithi073 2023-01-13T01:25:05.310 INFO:teuthology.kill:2023-01-13 01:25:05,309.309 INFO:teuthology.orchestra.console:Power off smithi084 2023-01-13T01:25:13.682 INFO:teuthology.kill:2023-01-13 01:25:13,681.681 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2023-01-13T01:25:13.823 INFO:teuthology.kill:2023-01-13 01:25:13,823.823 INFO:teuthology.lock.ops:unlocked: smithi084.front.sepia.ceph.com 2023-01-13T01:25:17.786 INFO:teuthology.kill:2023-01-13 01:25:17,786.786 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2023-01-13T01:25:17.938 INFO:teuthology.kill:2023-01-13 01:25:17,938.938 INFO:teuthology.lock.ops:unlocked: smithi073.front.sepia.ceph.com 2023-01-13T01:27:18.260 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-01-13T01:27:18.355 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi073 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-01-12_20:03:13-rados-pacific_16.2.11_RC6.6-distro-default-smithi/7138269 2023-01-13T01:27:18.356 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi084 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-01-12_20:03:13-rados-pacific_16.2.11_RC6.6-distro-default-smithi/7138268