2021-11-16T15:22:44.661 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-16T15:22:44.777 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging 2021-11-16T15:22:44.778 INFO:teuthology.lock.ops:Updating [ubuntu@smithi085.front.sepia.ceph.com]: reset os type and version on server 2021-11-16T15:22:44.778 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-11-16T15:22:44.796 INFO:teuthology.lock.ops:Node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging is complete 2021-11-16T15:22:44.797 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging 2021-11-16T15:22:44.797 INFO:teuthology.lock.ops:Updating [ubuntu@smithi170.front.sepia.ceph.com]: reset os type and version on server 2021-11-16T15:22:44.797 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2021-11-16T15:22:44.817 INFO:teuthology.lock.ops:Node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging is complete 2021-11-16T15:22:45.017 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of centos 8.stream 2021-11-16T15:22:45.033 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of centos 8.stream 2021-11-16T15:22:45.348 INFO:teuthology.orchestra.console:Power off smithi085 2021-11-16T15:22:45.363 INFO:teuthology.orchestra.console:Power off smithi170 2021-11-16T15:22:53.728 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-11-16T15:22:53.730 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-11-16T15:22:53.830 INFO:teuthology.orchestra.console:Power on smithi085 2021-11-16T15:22:53.840 INFO:teuthology.orchestra.console:Power on smithi170 2021-11-16T15:23:02.322 INFO:teuthology.orchestra.console:Power on for smithi170 completed 2021-11-16T15:23:02.327 INFO:teuthology.orchestra.console:Power on for smithi085 completed 2021-11-16T15:23:02.424 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish 2021-11-16T15:23:02.428 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish 2021-11-16T15:27:14.686 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-16T15:27:30.013 INFO:teuthology.provision.fog.smithi085:Node is ready 2021-11-16T15:27:30.074 INFO:teuthology.orchestra.run.smithi085.stdout:smithi047 2021-11-16T15:27:30.131 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.47 smithi047.front.sepia.ceph.com smithi047 2021-11-16T15:27:30.493 INFO:teuthology.provision.fog.smithi085:Deploy complete! 2021-11-16T15:27:52.122 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-16T15:27:52.388 INFO:teuthology.provision.fog.smithi170:Node is ready 2021-11-16T15:27:52.446 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com 2021-11-16T15:27:52.503 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170 2021-11-16T15:27:52.857 INFO:teuthology.provision.fog.smithi170:Deploy complete! 2021-11-16T15:27:53.014 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com 2021-11-16T15:27:53.015 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-16T15:27:53.034 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com 2021-11-16T15:27:53.034 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-16T15:27:53.057 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server 2021-11-16T15:27:53.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2021-11-16T15:27:53.294 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2021-11-16T15:27:53.350 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="CentOS Stream" 2021-11-16T15:27:53.350 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="8" 2021-11-16T15:27:53.350 INFO:teuthology.orchestra.run.smithi085.stdout:ID="centos" 2021-11-16T15:27:53.350 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE="rhel fedora" 2021-11-16T15:27:53.351 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="8" 2021-11-16T15:27:53.351 INFO:teuthology.orchestra.run.smithi085.stdout:PLATFORM_ID="platform:el8" 2021-11-16T15:27:53.351 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-16T15:27:53.351 INFO:teuthology.orchestra.run.smithi085.stdout:ANSI_COLOR="0;31" 2021-11-16T15:27:53.351 INFO:teuthology.orchestra.run.smithi085.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-16T15:27:53.352 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://centos.org/" 2021-11-16T15:27:53.352 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-16T15:27:53.352 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-16T15:27:53.352 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-16T15:27:53.354 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-11-16T15:27:53.371 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server 2021-11-16T15:27:53.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2021-11-16T15:27:53.565 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2021-11-16T15:27:53.622 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2021-11-16T15:27:53.623 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8" 2021-11-16T15:27:53.623 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2021-11-16T15:27:53.623 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2021-11-16T15:27:53.623 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8" 2021-11-16T15:27:53.623 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8" 2021-11-16T15:27:53.624 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-16T15:27:53.624 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2021-11-16T15:27:53.624 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-16T15:27:53.624 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2021-11-16T15:27:53.625 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-16T15:27:53.625 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-16T15:27:53.625 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-16T15:27:53.626 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2021-11-16T15:27:53.718 INFO:teuthology.dispatcher.supervisor:Running job 6506508 2021-11-16T15:27:53.718 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-16_13:07:14-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi/6506508 --name yuriw-2021-11-16_13:07:14-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2021-11-16_13:07:14-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi/6506508/orig.config.yaml 2021-11-16T15:27:53.722 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-11-16_13:07:14-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi/6506508 2021-11-16T15:27:53.723 INFO:teuthology.dispatcher.supervisor:Job PID: 18724 2021-11-16T15:27:53.724 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-17T03:28:48.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-17T03:28:48.336 INFO:teuthology.kill:Killing Pids: {18724} 2021-11-17T03:28:49.559 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-11-17T03:28:49.560 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-11-17T03:28:49.560 INFO:teuthology.misc:Compressing logs... 2021-11-17T03:28:49.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2021-11-17T03:28:49.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2021-11-17T03:28:49.800 INFO:teuthology.orchestra.run.smithi085.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-11-17T03:28:49.986 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2021-11-17T03:28:50.031 INFO:teuthology.orchestra.run.smithi085.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-11-17T03:28:50.031 INFO:teuthology.orchestra.run.smithi085.stderr:tar: Error is not recoverable: exiting now 2021-11-17T03:28:50.083 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2021-11-17T03:28:50.083 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now 2021-11-17T03:28:50.084 INFO:teuthology.misc:Compressing logs... 2021-11-17T03:28:50.441 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-11-17T03:28:50.646 INFO:teuthology.misc:Compressing logs... 2021-11-17T03:29:12.599 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2021-11-17T03:29:14.521 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-11-17T03:29:34.046 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-mon.smithi085.log: file size changed while zipping 2021-11-17T03:29:37.260 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-11-17T03:29:47.233 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-mon.smithi170.log: file size changed while zipping 2021-11-17T03:29:54.947 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-mgr.smithi170.cdknqy.log: file size changed while zipping 2021-11-17T03:29:59.716 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-11-17T03:30:16.307 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-11-17T03:30:20.075 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-11-17T03:30:41.743 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-11-17T03:30:49.757 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/2f3d120e-46f3-11ec-8c2c-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2021-11-17T03:31:37.900 INFO:teuthology.kill:No teuthology processes running 2021-11-17T03:31:39.355 INFO:teuthology.kill:Nuking machines: ['smithi085', 'smithi170'] 2021-11-17T03:31:40.339 INFO:teuthology.kill:2021-11-17 03:31:40,339.339 INFO:teuthology.nuke:targets: 2021-11-17T03:31:40.340 INFO:teuthology.kill: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDZpEKVDtfmx67sGiiMzfCMX/T22hJvX1QKp5q9I+LDqDCSTHuOJqTakqnt7BulVTf8YBTYKz+T/IhQIILGaxChUvojOXxI5x3SDRnaC8ehfshdVy6BPRdn7kD9lKGyC2VhYdghNYhZNPiUGjJ9XduCDJJU/UselldKFU0mQQNjE6GbwDmoPQJyY7jSCAmnsx8bcZJTif6pxXviWhFmtX20F7kqGOhcl+WBxpl/DTdPCtrA56fq4Q1RNBln/Xp8XUZ1MxuQ0JccAl8LCgXo7khuU8eQbtXNE+l2SepaGth4D0SBHyuCprGz7UH4rZi7vmjiz8vyQiWR6igXFYMfDjDZCcaqpglvCS9eOftiv1gXv+AovN+O2XNmYxsScOqFdLsfXTsj1lSGzp2hwQjTFBypfJQKqTGKh1MkWUEnO6blavij6Oenb9LkqmOzRb3T6b5A2bBW1kH12DXRxxy9Ft4T+2kBN3Y9aqQu05+vd/UQukbHmZi4QFByl3L6BS7mF8= 2021-11-17T03:31:40.340 INFO:teuthology.kill: smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8MATNeKPec33DA/jjeMKRrIMAM/Lb0wycoBqj5C5s/sVc/MHNUF83Cb8VWARzICUwTFisWENSmtUVCVVeTFaYZFKTuDxvP6OfeoEZMwbcGZUzKecq/D6+9wwGWJjb/Nm9fwFJGQTgnbny2fV4WTPNzyY8TzkrH4xdewB0n+TuzS5TJXSKSWZMoZOF7tgqbfjq+Culu13QffWsZXGqKLZQnnEhGhitZ9AhgT1zSv86fKdu5lm49ZKxQMYr3c4V11pBVhkqZzPSBgMFLpKfh2lNCfB0dZfqfnAits1gF+faSns0gd9+5OHtxSrWzD0nxXeuVxHL6LMpWJTg9VRJurCxj/BgXklCIswmYhqWsjyD8Es6kan/ksPjuFo22EnDr0+Pscuz/o+BwbMuTBxQDbmiYTqUoSwB0JWHaB5G29hWgHgQap5h1bOCRWuYiJhHG7fZEynJsNowxkrZY2ELEja2D8WNhw501jNOx5fiwBdQ8tjsR1TSG5iQOK2+49/gsBU= 2021-11-17T03:31:40.368 INFO:teuthology.kill:2021-11-17 03:31:40,368.368 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-17T03:31:40.371 INFO:teuthology.kill:2021-11-17 03:31:40,371.371 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-17T03:31:40.456 INFO:teuthology.kill:2021-11-17 03:31:40,456.456 INFO:teuthology.orchestra.console:Power off smithi085 2021-11-17T03:31:40.476 INFO:teuthology.kill:2021-11-17 03:31:40,474.474 INFO:teuthology.orchestra.console:Power off smithi170 2021-11-17T03:31:48.839 INFO:teuthology.kill:2021-11-17 03:31:48,839.839 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-11-17T03:31:48.995 INFO:teuthology.kill:2021-11-17 03:31:48,994.994 INFO:teuthology.lock.ops:unlocked: smithi085.front.sepia.ceph.com 2021-11-17T03:31:52.965 INFO:teuthology.kill:2021-11-17 03:31:52,964.964 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-11-17T03:31:53.116 INFO:teuthology.kill:2021-11-17 03:31:53,115.115 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com 2021-11-17T03:33:53.487 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-17T03:33:53.589 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi085 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-16_20:03:49-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6507847 2021-11-17T03:33:53.589 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi170 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-16_20:03:49-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6507846