2022-02-15T15:52:29.817 INFO:root:teuthology version: 1.1.0-7cebb3f2
2022-02-15T15:52:29.951 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging
2022-02-15T15:52:29.952 INFO:teuthology.lock.ops:Updating [ubuntu@smithi140.front.sepia.ceph.com]: reset os type and version on server
2022-02-15T15:52:29.952 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2022-02-15T15:52:29.971 INFO:teuthology.lock.ops:Node 'ubuntu@smithi140.front.sepia.ceph.com' reimaging is complete
2022-02-15T15:52:29.971 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi146.front.sepia.ceph.com' reimaging
2022-02-15T15:52:29.972 INFO:teuthology.lock.ops:Updating [ubuntu@smithi146.front.sepia.ceph.com]: reset os type and version on server
2022-02-15T15:52:29.972 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server
2022-02-15T15:52:29.993 INFO:teuthology.lock.ops:Node 'ubuntu@smithi146.front.sepia.ceph.com' reimaging is complete
2022-02-15T15:52:30.235 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of centos 8.stream
2022-02-15T15:52:30.252 INFO:teuthology.provision.fog.smithi146:Scheduling deploy of centos 8.stream
2022-02-15T15:52:30.603 INFO:teuthology.orchestra.console:Power off smithi140
2022-02-15T15:52:30.759 INFO:teuthology.orchestra.console:Power off smithi146
2022-02-15T15:52:38.971 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2022-02-15T15:52:39.073 INFO:teuthology.orchestra.console:Power on smithi140
2022-02-15T15:52:43.234 INFO:teuthology.orchestra.console:Power off for smithi146 completed
2022-02-15T15:52:43.335 INFO:teuthology.orchestra.console:Power on smithi146
2022-02-15T15:52:47.541 INFO:teuthology.orchestra.console:Power on for smithi140 completed
2022-02-15T15:52:47.584 INFO:teuthology.orchestra.console:Power on for smithi146 completed
2022-02-15T15:52:47.643 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2022-02-15T15:52:47.685 INFO:teuthology.provision.fog.smithi146:Waiting for deploy to finish
2022-02-15T15:57:18.201 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-15T15:57:28.424 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-15T15:57:28.735 INFO:teuthology.provision.fog.smithi140:Node is ready
2022-02-15T15:57:28.755 INFO:teuthology.orchestra.run.smithi140.stdout:smithi140.front.sepia.ceph.com
2022-02-15T15:57:28.815 INFO:teuthology.orchestra.run.smithi140.stdout:172.21.15.140 smithi140.front.sepia.ceph.com smithi140
2022-02-15T15:57:29.159 INFO:teuthology.provision.fog.smithi140:Deploy complete!
2022-02-15T15:57:43.205 INFO:teuthology.provision.fog.smithi146:Node is ready
2022-02-15T15:57:43.228 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146.front.sepia.ceph.com
2022-02-15T15:57:43.287 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146 smithi146.front.sepia.ceph.com smithi146
2022-02-15T15:57:43.661 INFO:teuthology.provision.fog.smithi146:Deploy complete!
2022-02-15T15:57:43.836 INFO:teuthology.lock.ops:Checking smithi140.front.sepia.ceph.com
2022-02-15T15:57:43.837 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-15T15:57:43.871 INFO:teuthology.lock.ops:Checking smithi146.front.sepia.ceph.com
2022-02-15T15:57:43.872 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-15T15:57:43.891 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: set os type and version on server
2022-02-15T15:57:43.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2022-02-15T15:57:44.137 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64
2022-02-15T15:57:44.194 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="CentOS Stream"
2022-02-15T15:57:44.194 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="8"
2022-02-15T15:57:44.195 INFO:teuthology.orchestra.run.smithi140.stdout:ID="centos"
2022-02-15T15:57:44.195 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE="rhel fedora"
2022-02-15T15:57:44.195 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="8"
2022-02-15T15:57:44.195 INFO:teuthology.orchestra.run.smithi140.stdout:PLATFORM_ID="platform:el8"
2022-02-15T15:57:44.196 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="CentOS Stream 8"
2022-02-15T15:57:44.196 INFO:teuthology.orchestra.run.smithi140.stdout:ANSI_COLOR="0;31"
2022-02-15T15:57:44.196 INFO:teuthology.orchestra.run.smithi140.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-02-15T15:57:44.196 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://centos.org/"
2022-02-15T15:57:44.196 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-15T15:57:44.197 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-15T15:57:44.197 INFO:teuthology.orchestra.run.smithi140.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-02-15T15:57:44.199 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2022-02-15T15:57:44.218 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: set os type and version on server
2022-02-15T15:57:44.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2022-02-15T15:57:44.408 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64
2022-02-15T15:57:44.465 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream"
2022-02-15T15:57:44.466 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8"
2022-02-15T15:57:44.466 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos"
2022-02-15T15:57:44.466 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora"
2022-02-15T15:57:44.467 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8"
2022-02-15T15:57:44.467 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8"
2022-02-15T15:57:44.467 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 8"
2022-02-15T15:57:44.467 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31"
2022-02-15T15:57:44.468 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-02-15T15:57:44.468 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/"
2022-02-15T15:57:44.468 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-15T15:57:44.468 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-15T15:57:44.469 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-02-15T15:57:44.469 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server
2022-02-15T15:57:44.582 INFO:teuthology.dispatcher.supervisor:Running job 6684954
2022-02-15T15:57:44.583 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_7cebb3f2319fd6c8340c0f7cd15a137e747fd32e/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2022-02-15_15:46:59-rados-wip-mgr-fixes-pacific-dhp-distro-basic-smithi/6684954 --name nojha-2022-02-15_15:46:59-rados-wip-mgr-fixes-pacific-dhp-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/nojha-2022-02-15_15:46:59-rados-wip-mgr-fixes-pacific-dhp-distro-basic-smithi/6684954/orig.config.yaml
2022-02-15T15:57:44.588 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2022-02-15_15:46:59-rados-wip-mgr-fixes-pacific-dhp-distro-basic-smithi/6684954
2022-02-15T15:57:44.588 INFO:teuthology.dispatcher.supervisor:Job PID: 4816
2022-02-15T15:57:44.589 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-15T22:28:13.163 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-15T22:28:13.384 INFO:teuthology.kill:Killing Pids: {4816}
2022-02-15T22:28:14.491 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-02-15T22:28:14.492 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-02-15T22:28:14.492 INFO:teuthology.misc:Compressing logs...
2022-02-15T22:28:14.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2022-02-15T22:28:14.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2022-02-15T22:28:14.734 INFO:teuthology.orchestra.run.smithi140.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-02-15T22:28:14.924 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-02-15T22:28:14.964 INFO:teuthology.orchestra.run.smithi140.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-02-15T22:28:14.964 INFO:teuthology.orchestra.run.smithi140.stderr:tar: Error is not recoverable: exiting now
2022-02-15T22:28:15.002 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-02-15T22:28:15.002 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2022-02-15T22:28:15.003 INFO:teuthology.misc:Compressing logs...
2022-02-15T22:28:15.307 INFO:teuthology.misc:Compressing logs...
2022-02-15T22:28:27.117 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-mon.smithi140.log: file size changed while zipping
2022-02-15T22:28:29.650 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-mgr.smithi140.jdmahn.log: file size changed while zipping
2022-02-15T22:28:42.098 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-02-15T22:28:45.441 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-mon.smithi146.log: file size changed while zipping
2022-02-15T22:28:53.140 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-02-15T22:29:10.303 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-02-15T22:29:18.219 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-02-15T22:29:38.453 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-02-15T22:29:40.337 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-02-15T22:30:02.865 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-02-15T22:30:05.766 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/f6d492c0-8e78-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-02-15T22:30:28.125 INFO:teuthology.orchestra.run.smithi140.stderr:tar: .: file changed as we read it
2022-02-15T22:30:49.879 INFO:teuthology.kill:No teuthology processes running
2022-02-15T22:30:51.028 INFO:teuthology.kill:Nuking machines: ['smithi140', 'smithi146']
2022-02-15T22:30:52.011 INFO:teuthology.kill:2022-02-15 22:30:52,010.010 INFO:teuthology.nuke:targets:
2022-02-15T22:30:52.011 INFO:teuthology.kill:  smithi140.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIGxRgLkJEyVyX1Rgpp+X5S7x+5R1i1JH9UdqsJuSg1ko4TLYpVgn0TcLJRUfE2Dfy1MnU0tThZ/bSeUHUFTE5mA65uARNa0+bl8MWQ091LxY7IYBtrIlItQmYoMC/xrHKYFUtiFf4/r7XU2E9/Iit82TozgIsd1rpRIw5gGWM/9q2+pCKttaW4Z78pb69ZOilz2KHObuw9Cm+yromc9UC0YAmc2Kcz8XX3tScc7f6sronXlBrTsTM4sVxCRNKFsFcF4X/5sSzZkuczyQX/66MgHmTXZroLv9pdlytAAYc5+G9S7b9GgUujEOms3kXtNzbjPYwk/+zapsBn//VPdpwjprcqedelIi3ed7ge9iG1tdME/2qoHM/eQTmqv9ttqpSPDVdc36KPIk3TT2fkPs8bTQ8XhVPGp2vR60RngAfsveZNagHJAjiqqfd+SOJnQyUGY6nd7ygmRtrVPz1qab4q2PB1qPCB1NhxgU6BOhm7/WXwxFiYdqhUrwX6R2767k=
2022-02-15T22:30:52.012 INFO:teuthology.kill:  smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFkeTsFE/SwIhsC9te72Cf48XxLWfW+76MiuHxk3OM/qnDy9wqHye9v1hcBQMmIpyEaI4jAJqskWlT7uAGtm1/CguXT1VdwClI7zhj0gXhD08WVsNEVBV9G70RXCSXGThdG1kkvNFJi+Qe+yKq2/+oznPDmgX+kQH8pp6J4dzxL6Tzfs2zlFMA/ouMoyHV4nCzJZAkYnOilNJ7k2hdBIiWS1Nzf/z0MYSofp2HcNgnrCK+giDapkVc+CHsTV99CAec/4r+44PTtp6ojk70j4zMEOCCoGTfRpz9LucpWKmrwVZ5v6YRK3HcRAt/mBXsFGXJW3y4Gi4OmzcvwLDtj/wR8Nwvd4AQDR775jBDjL2ooFncrsRbkW+PLsi1EvL4fQb0yWnMsMtl2bpQ92c0uplM54PJuYg23NrokP8LUiLH8exHFvTXVaIrjhV5U7+w0tbWob9HOtO7YNc6KVoTOSVbxE++2zJo+iTHcdEi6yUy8R/BZVy3AnwTxNc3HgaAZm0=
2022-02-15T22:30:52.037 INFO:teuthology.kill:2022-02-15 22:30:52,037.037 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-15T22:30:52.040 INFO:teuthology.kill:2022-02-15 22:30:52,040.040 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-15T22:30:52.116 INFO:teuthology.kill:2022-02-15 22:30:52,115.115 INFO:teuthology.orchestra.console:Power off smithi140
2022-02-15T22:30:52.130 INFO:teuthology.kill:2022-02-15 22:30:52,130.130 INFO:teuthology.orchestra.console:Power off smithi146
2022-02-15T22:31:00.485 INFO:teuthology.kill:2022-02-15 22:31:00,485.485 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2022-02-15T22:31:00.636 INFO:teuthology.kill:2022-02-15 22:31:00,635.635 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2022-02-15T22:31:04.604 INFO:teuthology.kill:2022-02-15 22:31:04,603.603 INFO:teuthology.orchestra.console:Power off for smithi146 completed
2022-02-15T22:31:04.747 INFO:teuthology.kill:2022-02-15 22:31:04,747.747 INFO:teuthology.lock.ops:unlocked: smithi146.front.sepia.ceph.com
2022-02-15T22:33:05.046 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-15T22:33:05.174 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi140 but it was locked by another job: /home/teuthworker/archive/mrfreezeex-2022-02-15_16:30:38-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6685305
2022-02-15T22:33:05.174 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi146 but it was locked by another job: /home/teuthworker/archive/mrfreezeex-2022-02-15_16:30:38-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6685305