2022-01-24T22:50:44.301 INFO:root:teuthology version: 1.1.0-7c0cb867
2022-01-24T22:50:44.432 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi052.front.sepia.ceph.com' reimaging
2022-01-24T22:50:44.434 INFO:teuthology.lock.ops:Updating [ubuntu@smithi052.front.sepia.ceph.com]: reset os type and version on server
2022-01-24T22:50:44.434 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server
2022-01-24T22:50:44.458 INFO:teuthology.lock.ops:Node 'ubuntu@smithi052.front.sepia.ceph.com' reimaging is complete
2022-01-24T22:50:44.458 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging
2022-01-24T22:50:44.459 INFO:teuthology.lock.ops:Updating [ubuntu@smithi072.front.sepia.ceph.com]: reset os type and version on server
2022-01-24T22:50:44.460 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server
2022-01-24T22:50:44.481 INFO:teuthology.lock.ops:Node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging is complete
2022-01-24T22:50:44.672 INFO:teuthology.provision.fog.smithi052:Scheduling deploy of centos 8.3
2022-01-24T22:50:44.695 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of centos 8.3
2022-01-24T22:50:45.017 INFO:teuthology.orchestra.console:Power off smithi052
2022-01-24T22:50:45.053 INFO:teuthology.orchestra.console:Power off smithi072
2022-01-24T22:50:45.161 INFO:teuthology.orchestra.console:Power off for smithi052 completed
2022-01-24T22:50:45.193 INFO:teuthology.orchestra.console:Power off for smithi072 completed
2022-01-24T22:50:45.263 INFO:teuthology.orchestra.console:Power on smithi052
2022-01-24T22:50:45.295 INFO:teuthology.orchestra.console:Power on smithi072
2022-01-24T22:50:53.632 INFO:teuthology.orchestra.console:Power on for smithi052 completed
2022-01-24T22:50:53.662 INFO:teuthology.orchestra.console:Power on for smithi072 completed
2022-01-24T22:50:53.734 INFO:teuthology.provision.fog.smithi052:Waiting for deploy to finish
2022-01-24T22:50:53.763 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish
2022-01-24T22:55:01.964 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-24T22:55:02.271 INFO:teuthology.provision.fog.smithi052:Node is ready
2022-01-24T22:55:02.287 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052.front.sepia.ceph.com
2022-01-24T22:55:02.343 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52 smithi052.front.sepia.ceph.com smithi052
2022-01-24T22:55:02.658 INFO:teuthology.provision.fog.smithi052:Deploy complete!
2022-01-24T22:55:56.634 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-24T22:55:57.001 INFO:teuthology.provision.fog.smithi072:Node is ready
2022-01-24T22:55:57.017 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072.front.sepia.ceph.com
2022-01-24T22:55:57.074 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072
2022-01-24T22:55:57.404 INFO:teuthology.provision.fog.smithi072:Deploy complete!
2022-01-24T22:55:57.551 INFO:teuthology.lock.ops:Checking smithi052.front.sepia.ceph.com
2022-01-24T22:55:57.551 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-24T22:55:57.574 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com
2022-01-24T22:55:57.574 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-24T22:55:57.593 INFO:teuthology.lock.ops:Updating [smithi052.front.sepia.ceph.com]: set os type and version on server
2022-01-24T22:55:57.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2022-01-24T22:55:57.869 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64
2022-01-24T22:55:57.926 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Linux"
2022-01-24T22:55:57.926 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="8"
2022-01-24T22:55:57.926 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos"
2022-01-24T22:55:57.927 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora"
2022-01-24T22:55:57.927 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="8"
2022-01-24T22:55:57.927 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el8"
2022-01-24T22:55:57.927 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Linux 8"
2022-01-24T22:55:57.927 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31"
2022-01-24T22:55:57.928 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-24T22:55:57.928 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/"
2022-01-24T22:55:57.928 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2022-01-24T22:55:57.928 INFO:teuthology.orchestra.run.smithi052.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2022-01-24T22:55:57.929 INFO:teuthology.orchestra.run.smithi052.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2022-01-24T22:55:57.930 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server
2022-01-24T22:55:57.948 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server
2022-01-24T22:55:57.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com'
2022-01-24T22:55:58.207 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64
2022-01-24T22:55:58.262 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Linux"
2022-01-24T22:55:58.263 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8"
2022-01-24T22:55:58.263 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos"
2022-01-24T22:55:58.263 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora"
2022-01-24T22:55:58.263 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8"
2022-01-24T22:55:58.264 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8"
2022-01-24T22:55:58.264 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Linux 8"
2022-01-24T22:55:58.264 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31"
2022-01-24T22:55:58.265 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-24T22:55:58.265 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/"
2022-01-24T22:55:58.266 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2022-01-24T22:55:58.266 INFO:teuthology.orchestra.run.smithi072.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2022-01-24T22:55:58.267 INFO:teuthology.orchestra.run.smithi072.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2022-01-24T22:55:58.268 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server
2022-01-24T22:55:58.424 INFO:teuthology.dispatcher.supervisor:Running job 6638257
2022-01-24T22:55:58.425 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-01-24_17:43:02-rados-wip-yuri2-testing-2022-01-21-0949-pacific-distro-default-smithi/6638257 --name yuriw-2022-01-24_17:43:02-rados-wip-yuri2-testing-2022-01-21-0949-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.3_container_tools_3.0 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2022-01-24_17:43:02-rados-wip-yuri2-testing-2022-01-21-0949-pacific-distro-default-smithi/6638257/orig.config.yaml
2022-01-24T22:55:58.430 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-01-24_17:43:02-rados-wip-yuri2-testing-2022-01-21-0949-pacific-distro-default-smithi/6638257
2022-01-24T22:55:58.431 INFO:teuthology.dispatcher.supervisor:Job PID: 8798
2022-01-24T22:55:58.432 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-01-25T05:26:28.223 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-01-25T05:26:28.407 INFO:teuthology.kill:Killing Pids: {8798}
2022-01-25T05:26:29.561 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-01-25T05:26:29.562 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-01-25T05:26:29.563 INFO:teuthology.misc:Compressing logs...
2022-01-25T05:26:29.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2022-01-25T05:26:29.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com'
2022-01-25T05:26:29.826 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-01-25T05:26:30.024 INFO:teuthology.orchestra.run.smithi072.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-01-25T05:26:30.067 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-01-25T05:26:30.067 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2022-01-25T05:26:30.108 INFO:teuthology.orchestra.run.smithi072.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-01-25T05:26:30.109 INFO:teuthology.orchestra.run.smithi072.stderr:tar: Error is not recoverable: exiting now
2022-01-25T05:26:30.110 INFO:teuthology.misc:Compressing logs...
2022-01-25T05:26:30.367 INFO:teuthology.misc:Compressing logs...
2022-01-25T05:26:35.191 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-mgr.smithi052.mjgnjx.log: file size changed while zipping
2022-01-25T05:26:53.545 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-01-25T05:26:56.991 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-01-25T05:27:08.583 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-mon.smithi052.log: file size changed while zipping
2022-01-25T05:27:13.424 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-01-25T05:27:17.177 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-mon.smithi072.log: file size changed while zipping
2022-01-25T05:27:27.379 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-01-25T05:27:35.087 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-01-25T05:27:39.727 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-01-25T05:27:51.326 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-01-25T05:28:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/1b096938-7d6a-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-01-25T05:28:20.263 INFO:teuthology.orchestra.run.smithi052.stderr:tar: .: file changed as we read it
2022-01-25T05:28:36.164 INFO:teuthology.kill:No teuthology processes running
2022-01-25T05:28:37.247 INFO:teuthology.kill:Nuking machines: ['smithi052', 'smithi072']
2022-01-25T05:28:38.214 INFO:teuthology.kill:2022-01-25 05:28:38,214.214 INFO:teuthology.nuke:targets:
2022-01-25T05:28:38.215 INFO:teuthology.kill:  smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8ypoGPiH1YFC29jCcAJK5pntj55v/gHu+Md3BkpZEhGpsuwS6RlEhmyuFyN6/qr97CIw1yX1kEXEZkXXLKqyxwTDWNurU7G85vjsEZuTvDIZhUugGNks7AVYEGjUtNaKoKpCkpaZF/0dIVJMaZxvXcNG5S5qcpBOqECbBO/8T2uCKc3PrqDajCRPo7Z1mB+uEL0sRyBdunM9SqkHjXnmObgHkU8alQL2CvZCK6s8X8x8+6a0FpXPULCXaxqsyySW7M60tiTR0KFZ4xIm/KQqU/sSz3H05+JvKV10tsn4yEnlN+XO8N7JYs9Dz8npc7QRH6yU3XJdMIeLc4FjCnnS0+YqKWdM8hggNRyCc2YJb8XVk81hlR1MHZaqGNqxhvuBxLi27yoe2rX8lkr8uEaN8S0T/SVdsXHvLYa5NAZgcE3k6AsYhNVfE5a3/CarODpu9l+showuvohlAjBenl9INk6xIEceGT+SkMAPMV3Mg7vQ4oTivubyIUNtVLin0yi0=
2022-01-25T05:28:38.215 INFO:teuthology.kill:  smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6OQTa5rLwfzZmtbr+WYVCBqYFPtlGgrAN2kkJJzhcSMSbNqkAeJWypVo9nz8j+nAlBQwUKBxHGVCIXygnn4BYUvXKwQrFKOpFajdNP1+CKN32Km4Sm9q+sm9Oc0qePuKECAa90frsvS23EvlOhnIekdjbxBbzjt4rHKdevmK270HFf6ZZONpT7v2ywN6xunyGyPbhgCIMt5DsVWy+HKXPUMPLXRe5XCGR53opzVsgGYV2DltjyI82oPbkVnCgwgGMtON4RqOe2xn0XHI+rXp1lZv34t1xEPv653f9cTgTNlibuK21ICac9Q2czti8s4mVSzcuQkVF6Nc0o6PjhnWMKCnmotsn/DDfcZ1+0W0q7ZUbL4+iFJlsStWg/ZGvJOUC1nXOrWLX4TbQik2VyQ4rLXPOEnLloTE5/7HrjnreDjjciTsetVdSn1xAYePBBCFVTdlSvNFfpeczKc1mhGBtZDFAUftSC3IilwX2bzW28QxJxcEeH6P6jtEw4klcluE=
2022-01-25T05:28:38.241 INFO:teuthology.kill:2022-01-25 05:28:38,241.241 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-25T05:28:38.244 INFO:teuthology.kill:2022-01-25 05:28:38,244.244 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-25T05:28:38.321 INFO:teuthology.kill:2022-01-25 05:28:38,320.320 INFO:teuthology.orchestra.console:Power off smithi072
2022-01-25T05:28:38.336 INFO:teuthology.kill:2022-01-25 05:28:38,336.336 INFO:teuthology.orchestra.console:Power off smithi052
2022-01-25T05:28:50.812 INFO:teuthology.kill:2022-01-25 05:28:50,811.811 INFO:teuthology.orchestra.console:Power off for smithi072 completed
2022-01-25T05:28:50.816 INFO:teuthology.kill:2022-01-25 05:28:50,816.816 INFO:teuthology.orchestra.console:Power off for smithi052 completed
2022-01-25T05:28:50.964 INFO:teuthology.kill:2022-01-25 05:28:50,963.963 INFO:teuthology.lock.ops:unlocked: smithi052.front.sepia.ceph.com
2022-01-25T05:28:50.966 INFO:teuthology.kill:2022-01-25 05:28:50,966.966 INFO:teuthology.lock.ops:unlocked: smithi072.front.sepia.ceph.com
2022-01-25T05:30:51.287 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-01-25T05:30:51.423 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi052 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-01-25_03:40:28-rados-wip-yuri4-testing-2022-01-24-1706-pacific-distro-default-smithi/6639208
2022-01-25T05:30:51.423 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi072 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-01-25_03:40:28-rados-wip-yuri4-testing-2022-01-24-1706-pacific-distro-default-smithi/6639208