2022-01-08T00:14:56.526 INFO:root:teuthology version: 1.1.0-95a7d479 2022-01-08T00:14:56.660 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi018.front.sepia.ceph.com' reimaging 2022-01-08T00:14:56.662 INFO:teuthology.lock.ops:Updating [ubuntu@smithi018.front.sepia.ceph.com]: reset os type and version on server 2022-01-08T00:14:56.662 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2022-01-08T00:14:56.684 INFO:teuthology.lock.ops:Node 'ubuntu@smithi018.front.sepia.ceph.com' reimaging is complete 2022-01-08T00:14:56.684 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi101.front.sepia.ceph.com' reimaging 2022-01-08T00:14:56.684 INFO:teuthology.lock.ops:Updating [ubuntu@smithi101.front.sepia.ceph.com]: reset os type and version on server 2022-01-08T00:14:56.685 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server 2022-01-08T00:14:56.705 INFO:teuthology.lock.ops:Node 'ubuntu@smithi101.front.sepia.ceph.com' reimaging is complete 2022-01-08T00:14:56.899 INFO:teuthology.provision.fog.smithi018:Scheduling deploy of centos 8.3 2022-01-08T00:14:56.915 INFO:teuthology.provision.fog.smithi101:Scheduling deploy of centos 8.3 2022-01-08T00:14:57.376 INFO:teuthology.orchestra.console:Power off smithi018 2022-01-08T00:14:57.393 INFO:teuthology.orchestra.console:Power off smithi101 2022-01-08T00:15:05.752 INFO:teuthology.orchestra.console:Power off for smithi018 completed 2022-01-08T00:15:05.854 INFO:teuthology.orchestra.console:Power on smithi018 2022-01-08T00:15:09.872 INFO:teuthology.orchestra.console:Power off for smithi101 completed 2022-01-08T00:15:09.974 INFO:teuthology.orchestra.console:Power on smithi101 2022-01-08T00:15:14.219 INFO:teuthology.orchestra.console:Power on for smithi018 completed 2022-01-08T00:15:14.320 INFO:teuthology.provision.fog.smithi018:Waiting for deploy to finish 2022-01-08T00:15:18.341 INFO:teuthology.orchestra.console:Power on for smithi101 completed 2022-01-08T00:15:18.442 INFO:teuthology.provision.fog.smithi101:Waiting for deploy to finish 2022-01-08T00:19:30.644 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-08T00:19:49.038 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-08T00:19:49.344 INFO:teuthology.provision.fog.smithi018:Node is ready 2022-01-08T00:19:49.359 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018.front.sepia.ceph.com 2022-01-08T00:19:49.414 INFO:teuthology.orchestra.run.smithi018.stdout:172.21.15.18 smithi018.front.sepia.ceph.com smithi018 2022-01-08T00:19:49.740 INFO:teuthology.provision.fog.smithi018:Deploy complete! 2022-01-08T00:19:51.299 INFO:teuthology.provision.fog.smithi101:Node is ready 2022-01-08T00:19:51.320 INFO:teuthology.orchestra.run.smithi101.stdout:smithi101.front.sepia.ceph.com 2022-01-08T00:19:51.376 INFO:teuthology.orchestra.run.smithi101.stdout:172.21.15.101 smithi101.front.sepia.ceph.com smithi101 2022-01-08T00:19:51.730 INFO:teuthology.provision.fog.smithi101:Deploy complete! 2022-01-08T00:19:51.880 INFO:teuthology.lock.ops:Checking smithi018.front.sepia.ceph.com 2022-01-08T00:19:51.881 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-08T00:19:51.903 INFO:teuthology.lock.ops:Checking smithi101.front.sepia.ceph.com 2022-01-08T00:19:51.903 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-08T00:19:51.922 INFO:teuthology.lock.ops:Updating [smithi018.front.sepia.ceph.com]: set os type and version on server 2022-01-08T00:19:51.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-01-08T00:19:52.139 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2022-01-08T00:19:52.197 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="CentOS Linux" 2022-01-08T00:19:52.198 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="8" 2022-01-08T00:19:52.198 INFO:teuthology.orchestra.run.smithi018.stdout:ID="centos" 2022-01-08T00:19:52.198 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE="rhel fedora" 2022-01-08T00:19:52.199 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="8" 2022-01-08T00:19:52.199 INFO:teuthology.orchestra.run.smithi018.stdout:PLATFORM_ID="platform:el8" 2022-01-08T00:19:52.199 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="CentOS Linux 8" 2022-01-08T00:19:52.199 INFO:teuthology.orchestra.run.smithi018.stdout:ANSI_COLOR="0;31" 2022-01-08T00:19:52.200 INFO:teuthology.orchestra.run.smithi018.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-08T00:19:52.200 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://centos.org/" 2022-01-08T00:19:52.200 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2022-01-08T00:19:52.200 INFO:teuthology.orchestra.run.smithi018.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2022-01-08T00:19:52.200 INFO:teuthology.orchestra.run.smithi018.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2022-01-08T00:19:52.202 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2022-01-08T00:19:52.220 INFO:teuthology.lock.ops:Updating [smithi101.front.sepia.ceph.com]: set os type and version on server 2022-01-08T00:19:52.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2022-01-08T00:19:52.475 INFO:teuthology.orchestra.run.smithi101.stdout:x86_64 2022-01-08T00:19:52.530 INFO:teuthology.orchestra.run.smithi101.stdout:NAME="CentOS Linux" 2022-01-08T00:19:52.530 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION="8" 2022-01-08T00:19:52.530 INFO:teuthology.orchestra.run.smithi101.stdout:ID="centos" 2022-01-08T00:19:52.531 INFO:teuthology.orchestra.run.smithi101.stdout:ID_LIKE="rhel fedora" 2022-01-08T00:19:52.531 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION_ID="8" 2022-01-08T00:19:52.531 INFO:teuthology.orchestra.run.smithi101.stdout:PLATFORM_ID="platform:el8" 2022-01-08T00:19:52.531 INFO:teuthology.orchestra.run.smithi101.stdout:PRETTY_NAME="CentOS Linux 8" 2022-01-08T00:19:52.532 INFO:teuthology.orchestra.run.smithi101.stdout:ANSI_COLOR="0;31" 2022-01-08T00:19:52.532 INFO:teuthology.orchestra.run.smithi101.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-08T00:19:52.532 INFO:teuthology.orchestra.run.smithi101.stdout:HOME_URL="https://centos.org/" 2022-01-08T00:19:52.532 INFO:teuthology.orchestra.run.smithi101.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2022-01-08T00:19:52.532 INFO:teuthology.orchestra.run.smithi101.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2022-01-08T00:19:52.533 INFO:teuthology.orchestra.run.smithi101.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2022-01-08T00:19:52.533 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server 2022-01-08T00:19:52.661 INFO:teuthology.dispatcher.supervisor:Running job 6601622 2022-01-08T00:19:52.662 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-01-07_20:51:42-rados-wip-yuri-testing-2022-01-07-0928-pacific-distro-default-smithi/6601622 --name yuriw-2022-01-07_20:51:42-rados-wip-yuri-testing-2022-01-07-0928-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.3_container_tools_3.0 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2022-01-07_20:51:42-rados-wip-yuri-testing-2022-01-07-0928-pacific-distro-default-smithi/6601622/orig.config.yaml 2022-01-08T00:19:52.665 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-01-07_20:51:42-rados-wip-yuri-testing-2022-01-07-0928-pacific-distro-default-smithi/6601622 2022-01-08T00:19:52.667 INFO:teuthology.dispatcher.supervisor:Job PID: 18384 2022-01-08T00:19:52.667 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-01-08T12:20:47.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-01-08T12:20:48.194 INFO:teuthology.kill:Killing Pids: {18384} 2022-01-08T12:20:49.361 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-01-08T12:20:49.362 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-01-08T12:20:49.362 INFO:teuthology.misc:Compressing logs... 2022-01-08T12:20:49.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-01-08T12:20:49.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2022-01-08T12:20:49.616 INFO:teuthology.orchestra.run.smithi018.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-01-08T12:20:49.814 INFO:teuthology.orchestra.run.smithi101.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-01-08T12:20:49.861 INFO:teuthology.orchestra.run.smithi018.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-01-08T12:20:49.862 INFO:teuthology.orchestra.run.smithi018.stderr:tar: Error is not recoverable: exiting now 2022-01-08T12:20:49.915 INFO:teuthology.orchestra.run.smithi101.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-01-08T12:20:49.915 INFO:teuthology.orchestra.run.smithi101.stderr:tar: Error is not recoverable: exiting now 2022-01-08T12:20:49.916 INFO:teuthology.misc:Compressing logs... 2022-01-08T12:20:50.241 INFO:teuthology.misc:Compressing logs... 2022-01-08T12:21:12.251 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-mon.smithi018.log: file size changed while zipping 2022-01-08T12:21:29.919 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2022-01-08T12:21:50.397 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-01-08T12:21:58.990 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-mgr.smithi018.mgavcy.log: file size changed while zipping 2022-01-08T12:22:04.674 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-01-08T12:22:10.947 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-mon.smithi101.log: file size changed while zipping 2022-01-08T12:22:31.477 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-01-08T12:22:41.842 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-01-08T12:22:55.473 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-01-08T12:23:10.255 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2022-01-08T12:23:32.822 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/07b756a4-701a-11ec-8c32-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-01-08T12:24:34.267 INFO:teuthology.kill:No teuthology processes running 2022-01-08T12:24:35.485 INFO:teuthology.kill:Nuking machines: ['smithi018', 'smithi101'] 2022-01-08T12:24:36.543 INFO:teuthology.kill:2022-01-08 12:24:36,542.542 INFO:teuthology.nuke:targets: 2022-01-08T12:24:36.543 INFO:teuthology.kill: smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0q6OmdM4J2s3Q9MF1S1H0XFL8aT/dIXNu/YMhSVUhaMOdUHcEPfo1VJtN2Ty0pRIJA++rJf2XI5wKoMi82D89rsA3v2gb/jIuoXDhyOQkfk6Lkq2yywifoC4fzx5Rl4TfsQPiVXbKNQN2x7lMSlpIbSWfeBGYpAIwJpqi3AG0PfGfefqPrr3ityn8mSVCUHWtEHX3CpAjOE73407Q33bN+HLcnMv3vMGIfH5U+J6N/l66Xo1mk+ydafzJ+0wl/i1okExOKC9+kFQfGvfFhztwaDK8jieZ4mo9CuXFZUsvn2q4f4d+JzFXLuCUY4/jZBzWhX7WkT/kT9SohTuEOMWhfgCRX/ts+53ddcPK011YYoHxZUsilqhiwSjpZgtU7OU0qS029cXFEwguR8fl/1ztOhBgCdyvEJGS+hVLHa3s8hfwR0V14qniILtpjXd6uDpeveyC6PuitnQV5bmXFJHsf9NbgxijBsrv9gNtOkhi9eMkF1wNWVXso8xjYhLsOME= 2022-01-08T12:24:36.544 INFO:teuthology.kill: smithi101.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGC3D3SqwQyyoh8EpOkuUTWlA/28VZXpiLp74Hvax59NuN1LUwL2W3EzmufaujxN9b1MkE04P4Shw9nq39VDW7QBMQRg6GnfFZy506T9F2OL0ZFnKcFNEGrzz8k02SDfQs48vHojTpDwLRiCTb6F6FpjTWdng0Uea8ONF7xNFzDDVDWd/BtFnAvyFI6xpomRou8XHyebj9sZ8qF9zIbNqEw/KmFKsHBcwDV1QONA/o2VWiFLRzHYKyR1cVczDYqC5HunlW5J0KQWFdCcYZZrRo1KYzy0tIGtHE2WQvOt21EhYM9FPg4gQKOhpgjFTzbh8KPmv48hJQB43O8GkVMYBQpnLuwL0tV/M9W9KH5XLgMeilDO9fotPlCve03dWBAsrJM48udxHan5Kp8b02e1H1cRMVK9UHeNc7luEjmi+Z/eIn1kWAf10XV7TrIBR+NeRr1dMLl6s8k0tMOIkP+7j1hNx40K0ka8rung+EbxMkKzwZ0+HPqpEvaq6Hi+wMxKc= 2022-01-08T12:24:36.569 INFO:teuthology.kill:2022-01-08 12:24:36,569.569 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-08T12:24:36.573 INFO:teuthology.kill:2022-01-08 12:24:36,572.572 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-08T12:24:36.649 INFO:teuthology.kill:2022-01-08 12:24:36,649.649 INFO:teuthology.orchestra.console:Power off smithi101 2022-01-08T12:24:36.661 INFO:teuthology.kill:2022-01-08 12:24:36,661.661 INFO:teuthology.orchestra.console:Power off smithi018 2022-01-08T12:24:45.019 INFO:teuthology.kill:2022-01-08 12:24:45,019.019 INFO:teuthology.orchestra.console:Power off for smithi101 completed 2022-01-08T12:24:45.166 INFO:teuthology.kill:2022-01-08 12:24:45,165.165 INFO:teuthology.lock.ops:unlocked: smithi101.front.sepia.ceph.com 2022-01-08T12:24:49.137 INFO:teuthology.kill:2022-01-08 12:24:49,136.136 INFO:teuthology.orchestra.console:Power off for smithi018 completed 2022-01-08T12:24:49.286 INFO:teuthology.kill:2022-01-08 12:24:49,286.286 INFO:teuthology.lock.ops:unlocked: smithi018.front.sepia.ceph.com 2022-01-08T12:26:49.597 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-01-08T12:26:49.694 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi018 but it was locked by another job: /home/teuthworker/archive/dis-2022-01-08_10:51:32-rbd-wip-dis-testing-distro-default-smithi/6602215 2022-01-08T12:26:49.694 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi101 but it was locked by another job: /home/teuthworker/archive/dis-2022-01-08_10:51:32-rbd-wip-dis-testing-distro-default-smithi/6602215