2022-02-06T16:06:35.057 INFO:root:teuthology version: 1.1.0-3094160c 2022-02-06T16:06:35.193 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi038.front.sepia.ceph.com' reimaging 2022-02-06T16:06:35.194 INFO:teuthology.lock.ops:Updating [ubuntu@smithi038.front.sepia.ceph.com]: reset os type and version on server 2022-02-06T16:06:35.194 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2022-02-06T16:06:35.212 INFO:teuthology.lock.ops:Node 'ubuntu@smithi038.front.sepia.ceph.com' reimaging is complete 2022-02-06T16:06:35.213 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi201.front.sepia.ceph.com' reimaging 2022-02-06T16:06:35.213 INFO:teuthology.lock.ops:Updating [ubuntu@smithi201.front.sepia.ceph.com]: reset os type and version on server 2022-02-06T16:06:35.214 INFO:teuthology.lock.ops:Updating smithi201.front.sepia.ceph.com on lock server 2022-02-06T16:06:35.234 INFO:teuthology.lock.ops:Node 'ubuntu@smithi201.front.sepia.ceph.com' reimaging is complete 2022-02-06T16:06:35.420 INFO:teuthology.provision.fog.smithi038:Scheduling deploy of centos 8.stream 2022-02-06T16:06:35.442 INFO:teuthology.provision.fog.smithi201:Scheduling deploy of centos 8.stream 2022-02-06T16:06:35.792 INFO:teuthology.orchestra.console:Power off smithi038 2022-02-06T16:06:35.833 INFO:teuthology.orchestra.console:Power off smithi201 2022-02-06T16:06:48.291 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2022-02-06T16:06:48.302 INFO:teuthology.orchestra.console:Power off for smithi201 completed 2022-02-06T16:06:48.393 INFO:teuthology.orchestra.console:Power on smithi038 2022-02-06T16:06:48.404 INFO:teuthology.orchestra.console:Power on smithi201 2022-02-06T16:06:52.648 INFO:teuthology.orchestra.console:Power on for smithi038 completed 2022-02-06T16:06:52.652 INFO:teuthology.orchestra.console:Power on for smithi201 completed 2022-02-06T16:06:52.750 INFO:teuthology.provision.fog.smithi038:Waiting for deploy to finish 2022-02-06T16:06:52.753 INFO:teuthology.provision.fog.smithi201:Waiting for deploy to finish 2022-02-06T16:10:53.277 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-06T16:11:16.186 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-06T16:11:16.437 INFO:teuthology.provision.fog.smithi038:Node is ready 2022-02-06T16:11:16.497 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038.front.sepia.ceph.com 2022-02-06T16:11:16.556 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38 smithi038.front.sepia.ceph.com smithi038 2022-02-06T16:11:16.907 INFO:teuthology.provision.fog.smithi038:Deploy complete! 2022-02-06T16:11:18.476 INFO:teuthology.provision.fog.smithi201:Node is ready 2022-02-06T16:11:18.497 INFO:teuthology.orchestra.run.smithi201.stdout:smithi201.front.sepia.ceph.com 2022-02-06T16:11:18.556 INFO:teuthology.orchestra.run.smithi201.stdout:172.21.15.201 smithi201.front.sepia.ceph.com smithi201 2022-02-06T16:11:18.927 INFO:teuthology.provision.fog.smithi201:Deploy complete! 2022-02-06T16:11:19.087 INFO:teuthology.lock.ops:Checking smithi038.front.sepia.ceph.com 2022-02-06T16:11:19.088 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-06T16:11:19.107 INFO:teuthology.lock.ops:Checking smithi201.front.sepia.ceph.com 2022-02-06T16:11:19.108 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-06T16:11:19.125 INFO:teuthology.lock.ops:Updating [smithi038.front.sepia.ceph.com]: set os type and version on server 2022-02-06T16:11:19.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2022-02-06T16:11:19.329 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2022-02-06T16:11:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2022-02-06T16:11:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="8" 2022-02-06T16:11:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2022-02-06T16:11:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2022-02-06T16:11:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="8" 2022-02-06T16:11:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el8" 2022-02-06T16:11:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-06T16:11:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2022-02-06T16:11:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-06T16:11:19.391 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2022-02-06T16:11:19.391 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-06T16:11:19.391 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-06T16:11:19.391 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-06T16:11:19.393 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2022-02-06T16:11:19.411 INFO:teuthology.lock.ops:Updating [smithi201.front.sepia.ceph.com]: set os type and version on server 2022-02-06T16:11:19.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com' 2022-02-06T16:11:19.599 INFO:teuthology.orchestra.run.smithi201.stdout:x86_64 2022-02-06T16:11:19.657 INFO:teuthology.orchestra.run.smithi201.stdout:NAME="CentOS Stream" 2022-02-06T16:11:19.657 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION="8" 2022-02-06T16:11:19.657 INFO:teuthology.orchestra.run.smithi201.stdout:ID="centos" 2022-02-06T16:11:19.658 INFO:teuthology.orchestra.run.smithi201.stdout:ID_LIKE="rhel fedora" 2022-02-06T16:11:19.658 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION_ID="8" 2022-02-06T16:11:19.658 INFO:teuthology.orchestra.run.smithi201.stdout:PLATFORM_ID="platform:el8" 2022-02-06T16:11:19.658 INFO:teuthology.orchestra.run.smithi201.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-06T16:11:19.659 INFO:teuthology.orchestra.run.smithi201.stdout:ANSI_COLOR="0;31" 2022-02-06T16:11:19.659 INFO:teuthology.orchestra.run.smithi201.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-06T16:11:19.659 INFO:teuthology.orchestra.run.smithi201.stdout:HOME_URL="https://centos.org/" 2022-02-06T16:11:19.660 INFO:teuthology.orchestra.run.smithi201.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-06T16:11:19.660 INFO:teuthology.orchestra.run.smithi201.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-06T16:11:19.661 INFO:teuthology.orchestra.run.smithi201.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-06T16:11:19.661 INFO:teuthology.lock.ops:Updating smithi201.front.sepia.ceph.com on lock server 2022-02-06T16:11:19.760 INFO:teuthology.dispatcher.supervisor:Running job 6665046 2022-02-06T16:11:19.821 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-02-06_16:04:09-rados-wip-yuri2-testing-2022-02-04-1646-pacific-distro-default-smithi/6665046 --name yuriw-2022-02-06_16:04:09-rados-wip-yuri2-testing-2022-02-04-1646-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2022-02-06_16:04:09-rados-wip-yuri2-testing-2022-02-04-1646-pacific-distro-default-smithi/6665046/orig.config.yaml 2022-02-06T16:11:19.825 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-02-06_16:04:09-rados-wip-yuri2-testing-2022-02-04-1646-pacific-distro-default-smithi/6665046 2022-02-06T16:11:19.826 INFO:teuthology.dispatcher.supervisor:Job PID: 16176 2022-02-06T16:11:19.827 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-06T22:41:48.292 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-06T22:41:48.475 INFO:teuthology.kill:Killing Pids: {16176} 2022-02-06T22:41:49.723 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-02-06T22:41:49.724 INFO:teuthology.task.internal:roles: ubuntu@smithi201.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-02-06T22:41:49.724 INFO:teuthology.misc:Compressing logs... 2022-02-06T22:41:49.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2022-02-06T22:41:49.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com' 2022-02-06T22:41:49.970 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-02-06T22:41:50.160 INFO:teuthology.orchestra.run.smithi201.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-02-06T22:41:50.211 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-02-06T22:41:50.211 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2022-02-06T22:41:50.258 INFO:teuthology.orchestra.run.smithi201.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-02-06T22:41:50.259 INFO:teuthology.orchestra.run.smithi201.stderr:tar: Error is not recoverable: exiting now 2022-02-06T22:41:50.260 INFO:teuthology.misc:Compressing logs... 2022-02-06T22:41:50.683 INFO:teuthology.misc:Compressing logs... 2022-02-06T22:42:07.276 INFO:teuthology.orchestra.run.smithi038.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-02-06T22:42:07.611 INFO:teuthology.orchestra.run.smithi201.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-02-06T22:42:19.995 INFO:teuthology.orchestra.run.smithi038.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-mon.smithi038.log: file size changed while zipping 2022-02-06T22:42:27.666 INFO:teuthology.orchestra.run.smithi201.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2022-02-06T22:42:31.063 INFO:teuthology.orchestra.run.smithi201.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-mon.smithi201.log: file size changed while zipping 2022-02-06T22:42:31.154 INFO:teuthology.orchestra.run.smithi201.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-mgr.smithi201.rjopwt.log: file size changed while zipping 2022-02-06T22:42:31.358 INFO:teuthology.orchestra.run.smithi038.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-02-06T22:42:45.515 INFO:teuthology.orchestra.run.smithi201.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2022-02-06T22:42:51.391 INFO:teuthology.orchestra.run.smithi038.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-02-06T22:42:54.404 INFO:teuthology.orchestra.run.smithi038.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-mgr.smithi038.cntvlb.log: file size changed while zipping 2022-02-06T22:43:01.724 INFO:teuthology.orchestra.run.smithi201.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-02-06T22:43:13.116 INFO:teuthology.orchestra.run.smithi038.stderr:gzip: /var/log/ceph/db888f84-8767-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-02-06T22:43:45.057 INFO:teuthology.kill:No teuthology processes running 2022-02-06T22:43:46.253 INFO:teuthology.kill:Nuking machines: ['smithi038', 'smithi201'] 2022-02-06T22:43:47.208 INFO:teuthology.kill:2022-02-06 22:43:47,208.208 INFO:teuthology.nuke:targets: 2022-02-06T22:43:47.209 INFO:teuthology.kill: smithi038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHIGNcQqoFzr+WCYjfcUBYfwIx4mpvXPBiR0TqwlNeg45KfDtIeEdkV2x6NW8UepYcspfhwItpg6DR9j9ZIBoKC5PLumziSUAQfzBNcGdyobKe8erDj7tPnmw6ggfSWoTk3O2E9Vs7qtZCYtc3aK/H2v4WRzLG3yltGIBCWcrlr9QbODl+NDY6DgBAqbm7BMnvBaSn7/O4NfJsgkoH/7zMqPPY6pXnNx3ZhNIUThC+aunosK5PNTaMLGnComDfeDj/YUmJsnbNosSQP4+Ti7Ss29Acu05WpQ78KuSDvWl6pqD4LAoPZ2UJCMpsS3Y0tgyj14ws5es8WfX6FbSk5NuUumyR64fmmoOS6SOn+qOW8FTBhfoDfL+8rqlJLtlDqmULNVnZiJnIaNvvojJCyH4CxB67qnSpkSJPSpdMJcxFYbmQbIbNYVa1YXLFtYljOQfgjyNvlQBhO61iN41dtK5woIMvUPgWsMO8ooHrqQaVqYqqQ9rE0zxShPv6wUH0ZRE= 2022-02-06T22:43:47.209 INFO:teuthology.kill: smithi201.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDlVRyOEmDatn+B6XOFzu7nI5PW9dIcxWs+vjxo4dckzPSe4Dwzv8+Buaxv9FbzLyHScDMpF1pnaBH8CGT9rM6vFH0eumoC7sfcb8cwvs7gkpnBULcK6IQLlbZAvahoUMv6uoMWLCggl6MUjFybMO2P8ZTejNJL71ZS8tysqg2skRQGPc43Km4d6/sBDqTSKA8bJ9OeINAn/BPe8JZV+/uMU7cZdYAqF9D2dVUt2cd06hVA2EOoiEldL1ZBIJK6NwpQELOgvpgxHHKt6UeUgHpSXBBrySxj65Ji/zjcV46bSZeFi0BTMQxumcvcwN+cDMBNLfspWkrwAMy1cSA/airT2i/lfDUhfW+R9C86rSuXSgvIbeXiLLU2aGSJfGZsLS/cdalRaR8uMfthYwM14paVlv7zJbc1+CohbIogkA07wwPwOiI/97DERAHG/Jo87JK2BUGMKXakaMdUJXP5aMTxmb1CEqZb+idUuh5i8Uouzy1Lo80taJJlfqV/mH0GMU= 2022-02-06T22:43:47.235 INFO:teuthology.kill:2022-02-06 22:43:47,234.234 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-06T22:43:47.237 INFO:teuthology.kill:2022-02-06 22:43:47,237.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-06T22:43:47.314 INFO:teuthology.kill:2022-02-06 22:43:47,314.314 INFO:teuthology.orchestra.console:Power off smithi201 2022-02-06T22:43:47.328 INFO:teuthology.kill:2022-02-06 22:43:47,328.328 INFO:teuthology.orchestra.console:Power off smithi038 2022-02-06T22:43:55.708 INFO:teuthology.kill:2022-02-06 22:43:55,707.707 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2022-02-06T22:43:55.712 INFO:teuthology.kill:2022-02-06 22:43:55,712.712 INFO:teuthology.orchestra.console:Power off for smithi201 completed 2022-02-06T22:43:55.872 INFO:teuthology.kill:2022-02-06 22:43:55,872.872 INFO:teuthology.lock.ops:unlocked: smithi038.front.sepia.ceph.com 2022-02-06T22:43:55.876 INFO:teuthology.kill:2022-02-06 22:43:55,876.876 INFO:teuthology.lock.ops:unlocked: smithi201.front.sepia.ceph.com 2022-02-06T22:45:56.180 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15