2022-02-15T16:00:15.353 INFO:root:teuthology version: 1.1.0-7cebb3f2
2022-02-15T16:00:15.475 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging
2022-02-15T16:00:15.476 INFO:teuthology.lock.ops:Updating [ubuntu@smithi187.front.sepia.ceph.com]: reset os type and version on server
2022-02-15T16:00:15.477 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server
2022-02-15T16:00:15.497 INFO:teuthology.lock.ops:Node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging is complete
2022-02-15T16:00:15.498 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi198.front.sepia.ceph.com' reimaging
2022-02-15T16:00:15.498 INFO:teuthology.lock.ops:Updating [ubuntu@smithi198.front.sepia.ceph.com]: reset os type and version on server
2022-02-15T16:00:15.498 INFO:teuthology.lock.ops:Updating smithi198.front.sepia.ceph.com on lock server
2022-02-15T16:00:15.519 INFO:teuthology.lock.ops:Node 'ubuntu@smithi198.front.sepia.ceph.com' reimaging is complete
2022-02-15T16:00:15.717 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of centos 8.stream
2022-02-15T16:00:15.732 INFO:teuthology.provision.fog.smithi198:Scheduling deploy of centos 8.stream
2022-02-15T16:00:16.113 INFO:teuthology.orchestra.console:Power off smithi198
2022-02-15T16:00:16.154 INFO:teuthology.orchestra.console:Power off smithi187
2022-02-15T16:00:24.490 INFO:teuthology.orchestra.console:Power off for smithi198 completed
2022-02-15T16:00:24.592 INFO:teuthology.orchestra.console:Power on smithi198
2022-02-15T16:00:28.631 INFO:teuthology.orchestra.console:Power off for smithi187 completed
2022-02-15T16:00:28.734 INFO:teuthology.orchestra.console:Power on smithi187
2022-02-15T16:00:32.977 INFO:teuthology.orchestra.console:Power on for smithi198 completed
2022-02-15T16:00:33.079 INFO:teuthology.provision.fog.smithi198:Waiting for deploy to finish
2022-02-15T16:00:37.101 INFO:teuthology.orchestra.console:Power on for smithi187 completed
2022-02-15T16:00:37.202 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish
2022-02-15T16:04:31.384 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-15T16:04:35.129 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-15T16:04:56.562 INFO:teuthology.provision.fog.smithi198:Node is ready
2022-02-15T16:04:56.585 INFO:teuthology.orchestra.run.smithi198.stdout:smithi198.front.sepia.ceph.com
2022-02-15T16:04:56.644 INFO:teuthology.orchestra.run.smithi198.stdout:172.21.15.198 smithi198.front.sepia.ceph.com smithi198
2022-02-15T16:04:57.016 INFO:teuthology.provision.fog.smithi198:Deploy complete!
2022-02-15T16:05:00.671 INFO:teuthology.provision.fog.smithi187:Node is ready
2022-02-15T16:05:00.693 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com
2022-02-15T16:05:00.752 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187
2022-02-15T16:05:01.129 INFO:teuthology.provision.fog.smithi187:Deploy complete!
2022-02-15T16:05:01.277 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com
2022-02-15T16:05:01.278 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-15T16:05:01.297 INFO:teuthology.lock.ops:Checking smithi198.front.sepia.ceph.com
2022-02-15T16:05:01.297 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-15T16:05:01.316 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server
2022-02-15T16:05:01.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2022-02-15T16:05:01.510 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64
2022-02-15T16:05:01.569 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream"
2022-02-15T16:05:01.569 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8"
2022-02-15T16:05:01.570 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos"
2022-02-15T16:05:01.571 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora"
2022-02-15T16:05:01.571 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8"
2022-02-15T16:05:01.572 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8"
2022-02-15T16:05:01.572 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 8"
2022-02-15T16:05:01.573 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31"
2022-02-15T16:05:01.573 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-02-15T16:05:01.574 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/"
2022-02-15T16:05:01.574 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-15T16:05:01.575 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-15T16:05:01.575 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-02-15T16:05:01.577 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server
2022-02-15T16:05:01.600 INFO:teuthology.lock.ops:Updating [smithi198.front.sepia.ceph.com]: set os type and version on server
2022-02-15T16:05:01.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi198.front.sepia.ceph.com'
2022-02-15T16:05:01.800 INFO:teuthology.orchestra.run.smithi198.stdout:x86_64
2022-02-15T16:05:01.858 INFO:teuthology.orchestra.run.smithi198.stdout:NAME="CentOS Stream"
2022-02-15T16:05:01.859 INFO:teuthology.orchestra.run.smithi198.stdout:VERSION="8"
2022-02-15T16:05:01.859 INFO:teuthology.orchestra.run.smithi198.stdout:ID="centos"
2022-02-15T16:05:01.859 INFO:teuthology.orchestra.run.smithi198.stdout:ID_LIKE="rhel fedora"
2022-02-15T16:05:01.859 INFO:teuthology.orchestra.run.smithi198.stdout:VERSION_ID="8"
2022-02-15T16:05:01.859 INFO:teuthology.orchestra.run.smithi198.stdout:PLATFORM_ID="platform:el8"
2022-02-15T16:05:01.860 INFO:teuthology.orchestra.run.smithi198.stdout:PRETTY_NAME="CentOS Stream 8"
2022-02-15T16:05:01.860 INFO:teuthology.orchestra.run.smithi198.stdout:ANSI_COLOR="0;31"
2022-02-15T16:05:01.860 INFO:teuthology.orchestra.run.smithi198.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-02-15T16:05:01.860 INFO:teuthology.orchestra.run.smithi198.stdout:HOME_URL="https://centos.org/"
2022-02-15T16:05:01.860 INFO:teuthology.orchestra.run.smithi198.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-15T16:05:01.861 INFO:teuthology.orchestra.run.smithi198.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-15T16:05:01.861 INFO:teuthology.orchestra.run.smithi198.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-02-15T16:05:01.861 INFO:teuthology.lock.ops:Updating smithi198.front.sepia.ceph.com on lock server
2022-02-15T16:05:01.956 INFO:teuthology.dispatcher.supervisor:Running job 6684968
2022-02-15T16:05:01.956 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_7cebb3f2319fd6c8340c0f7cd15a137e747fd32e/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2022-02-15_15:46:59-rados-wip-mgr-fixes-pacific-dhp-distro-basic-smithi/6684968 --name nojha-2022-02-15_15:46:59-rados-wip-mgr-fixes-pacific-dhp-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/nojha-2022-02-15_15:46:59-rados-wip-mgr-fixes-pacific-dhp-distro-basic-smithi/6684968/orig.config.yaml
2022-02-15T16:05:01.960 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2022-02-15_15:46:59-rados-wip-mgr-fixes-pacific-dhp-distro-basic-smithi/6684968
2022-02-15T16:05:01.960 INFO:teuthology.dispatcher.supervisor:Job PID: 6820
2022-02-15T16:05:01.961 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-15T22:35:31.175 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-15T22:35:31.341 INFO:teuthology.kill:Killing Pids: {6820}
2022-02-15T22:35:32.426 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-02-15T22:35:32.427 INFO:teuthology.task.internal:roles: ubuntu@smithi198.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-02-15T22:35:32.427 INFO:teuthology.misc:Compressing logs...
2022-02-15T22:35:32.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2022-02-15T22:35:32.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi198.front.sepia.ceph.com'
2022-02-15T22:35:32.655 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-02-15T22:35:32.845 INFO:teuthology.orchestra.run.smithi198.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-02-15T22:35:32.886 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-02-15T22:35:32.886 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now
2022-02-15T22:35:32.924 INFO:teuthology.orchestra.run.smithi198.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-02-15T22:35:32.924 INFO:teuthology.orchestra.run.smithi198.stderr:tar: Error is not recoverable: exiting now
2022-02-15T22:35:32.925 INFO:teuthology.misc:Compressing logs...
2022-02-15T22:35:33.221 INFO:teuthology.misc:Compressing logs...
2022-02-15T22:35:44.624 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-mon.smithi187.log: file size changed while zipping
2022-02-15T22:35:58.633 INFO:teuthology.orchestra.run.smithi198.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-02-15T22:36:06.038 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-02-15T22:36:21.711 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-02-15T22:36:29.049 INFO:teuthology.orchestra.run.smithi198.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-02-15T22:36:46.951 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-02-15T22:36:49.426 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-mgr.smithi187.myzpir.log: file size changed while zipping
2022-02-15T22:36:50.076 INFO:teuthology.orchestra.run.smithi198.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-02-15T22:36:53.461 INFO:teuthology.orchestra.run.smithi198.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-mon.smithi198.log: file size changed while zipping
2022-02-15T22:37:14.360 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-02-15T22:37:14.650 INFO:teuthology.orchestra.run.smithi198.stderr:gzip: /var/log/ceph/e74ef1c8-8e79-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-02-15T22:37:56.284 INFO:teuthology.kill:No teuthology processes running
2022-02-15T22:37:57.387 INFO:teuthology.kill:Nuking machines: ['smithi187', 'smithi198']
2022-02-15T22:37:58.354 INFO:teuthology.kill:2022-02-15 22:37:58,353.353 INFO:teuthology.nuke:targets:
2022-02-15T22:37:58.355 INFO:teuthology.kill:  smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIywd4YvzkKagp+ik0hwk8XMxK5wycniE5Q6f4Ga11rJrTqjCsmEd+NMUG6Swqfj6gCZOuE//BlaZ+WjxWMG8C/JPlzMlmbdVCxE68x1na6JprwbxNZunGJDdInkGDqiu0szfuUAzqaGe+So01/dyJoLhfi58neCSTRBjSPl6VUtLdoBDMEP+saM14/PRJdL+Oe3Nck5IuJ0qRLbM/h/NPHkSNfTTn+LGDWPtadgOKzKckGmNQ17Nzm3Debre0QS91g2JQMglMyUQKewqrn+UOixp/1oQKwET841TYzTmpS4QOeeVYf33NXubzaTNSuPGuq+O0WpuXcAoxHrHwKvQY1KEEUWXhEA6UgBljBJ2dgceJqw5ecG+MPFv1CtcVMRnDT3qJTp1N1MqeNCm+Ygg7n9FIFk1ix3bciUnSjTYgWJYeoFwbQ4x1WNv/wM7r0czzaZQcGRTSPRvMpJrFDSTFxzZS0myqU8CMYM4lNJ6GqhVFcPQC2npFeH0BqQsmYIM=
2022-02-15T22:37:58.355 INFO:teuthology.kill:  smithi198.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGHHEarh7aDE/MRTNZjnD0rRk2I8wA4hiwBmX6H024g51uo4+iNVZ4NzAeTosZeCW1LUPn+mi8pZanJ1rzdicUA6RHR7eq1f/AJs19KHUIzy2V3ujTWmaA25yNIT9hf6TV4mrH6el5q2KNyLClSZ51zWJiJju1Xkux4leyi1DIDFZqCnC1lujB1dWZuJmXLz87SinbwvWOGRAd0DMCbafPyUPq1JSj/VedKOMtjtcPQZ8GcWQFMaTp//3LBOwMg4WeQmSrbTsJ/Op6fW4E0D7Z2oMGD85vXZc+3hb50l2hgBHGfXGXvazbMckFWDY4Bfr9DC9WEmOBfK47oLPRADaQOKx4n6cCL7bMrcwhWLCsZgtCoQUJjVKEyx7VTXJ+BrgKlX9czywu+MmCuoxRUoTHWUdUVWgHgZFD0a5dNojqp92xjJAKqbdb0yW9oY/zeJ2paLDNueMSoeC0QN3X53Bd6C9sd2c0PsxbiRNoQswwbiRsRBhT5Ov7M/FSxHQQDx0=
2022-02-15T22:37:58.380 INFO:teuthology.kill:2022-02-15 22:37:58,380.380 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-15T22:37:58.386 INFO:teuthology.kill:2022-02-15 22:37:58,386.386 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-15T22:37:58.467 INFO:teuthology.kill:2022-02-15 22:37:58,466.466 INFO:teuthology.orchestra.console:Power off smithi198
2022-02-15T22:37:58.481 INFO:teuthology.kill:2022-02-15 22:37:58,481.481 INFO:teuthology.orchestra.console:Power off smithi187
2022-02-15T22:38:06.843 INFO:teuthology.kill:2022-02-15 22:38:06,842.842 INFO:teuthology.orchestra.console:Power off for smithi187 completed
2022-02-15T22:38:06.990 INFO:teuthology.kill:2022-02-15 22:38:06,989.989 INFO:teuthology.lock.ops:unlocked: smithi187.front.sepia.ceph.com
2022-02-15T22:38:10.943 INFO:teuthology.kill:2022-02-15 22:38:10,943.943 INFO:teuthology.orchestra.console:Power off for smithi198 completed
2022-02-15T22:38:11.089 INFO:teuthology.kill:2022-02-15 22:38:11,089.089 INFO:teuthology.lock.ops:unlocked: smithi198.front.sepia.ceph.com
2022-02-15T22:40:11.427 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-15T22:40:11.526 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi187 but it was locked by another job: /home/teuthworker/archive/adking-2022-02-15_22:33:11-orch:cephadm-wip-adk2-testing-2022-02-15-1304-pacific-distro-basic-smithi/6685829
2022-02-15T22:40:11.526 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi198 but it was locked by another job: /home/teuthworker/archive/adking-2022-02-15_22:33:11-orch:cephadm-wip-adk2-testing-2022-02-15-1304-pacific-distro-basic-smithi/6685829