2021-10-28T20:10:26.567 INFO:root:teuthology version: 1.1.0-c56135d1
2021-10-28T20:10:26.685 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging
2021-10-28T20:10:26.686 INFO:teuthology.lock.ops:Updating [ubuntu@smithi164.front.sepia.ceph.com]: reset os type and version on server
2021-10-28T20:10:26.686 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2021-10-28T20:10:26.704 INFO:teuthology.lock.ops:Node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging is complete
2021-10-28T20:10:26.705 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging
2021-10-28T20:10:26.705 INFO:teuthology.lock.ops:Updating [ubuntu@smithi170.front.sepia.ceph.com]: reset os type and version on server
2021-10-28T20:10:26.705 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2021-10-28T20:10:26.723 INFO:teuthology.lock.ops:Node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging is complete
2021-10-28T20:10:26.945 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of ubuntu 20.04
2021-10-28T20:10:26.972 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of ubuntu 20.04
2021-10-28T20:10:27.726 INFO:teuthology.orchestra.console:Power off smithi170
2021-10-28T20:10:27.831 INFO:teuthology.orchestra.console:Power off smithi164
2021-10-28T20:10:36.102 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2021-10-28T20:10:36.199 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2021-10-28T20:10:36.204 INFO:teuthology.orchestra.console:Power on smithi170
2021-10-28T20:10:36.300 INFO:teuthology.orchestra.console:Power on smithi164
2021-10-28T20:10:44.569 INFO:teuthology.orchestra.console:Power on for smithi170 completed
2021-10-28T20:10:44.658 INFO:teuthology.orchestra.console:Power on for smithi164 completed
2021-10-28T20:10:44.671 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish
2021-10-28T20:10:44.760 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish
2021-10-28T20:16:35.449 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-10-28T20:16:36.055 INFO:teuthology.provision.fog.smithi170:Node is ready
2021-10-28T20:16:36.061 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com
2021-10-28T20:16:36.068 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170
2021-10-28T20:16:36.257 INFO:teuthology.provision.fog.smithi170:Deploy complete!
2021-10-28T20:18:40.516 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-10-28T20:18:41.106 INFO:teuthology.provision.fog.smithi164:Node is ready
2021-10-28T20:18:41.113 INFO:teuthology.orchestra.run.smithi164.stdout:smithi164.front.sepia.ceph.com
2021-10-28T20:18:41.120 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.164 smithi164.front.sepia.ceph.com smithi164
2021-10-28T20:18:41.285 INFO:teuthology.provision.fog.smithi164:Deploy complete!
2021-10-28T20:18:41.470 INFO:teuthology.lock.ops:Checking smithi164.front.sepia.ceph.com
2021-10-28T20:18:41.470 INFO:teuthology.lock.ops:New key found. Updating...
2021-10-28T20:18:41.490 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com
2021-10-28T20:18:41.491 INFO:teuthology.lock.ops:New key found. Updating...
2021-10-28T20:18:41.509 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: set os type and version on server
2021-10-28T20:18:41.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com'
2021-10-28T20:18:41.678 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64
2021-10-28T20:18:41.728 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Ubuntu"
2021-10-28T20:18:41.729 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2021-10-28T20:18:41.729 INFO:teuthology.orchestra.run.smithi164.stdout:ID=ubuntu
2021-10-28T20:18:41.729 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE=debian
2021-10-28T20:18:41.729 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2021-10-28T20:18:41.730 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="20.04"
2021-10-28T20:18:41.730 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.ubuntu.com/"
2021-10-28T20:18:41.730 INFO:teuthology.orchestra.run.smithi164.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-10-28T20:18:41.730 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-10-28T20:18:41.730 INFO:teuthology.orchestra.run.smithi164.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-10-28T20:18:41.731 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_CODENAME=focal
2021-10-28T20:18:41.731 INFO:teuthology.orchestra.run.smithi164.stdout:UBUNTU_CODENAME=focal
2021-10-28T20:18:41.732 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2021-10-28T20:18:41.750 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server
2021-10-28T20:18:41.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2021-10-28T20:18:41.966 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64
2021-10-28T20:18:41.974 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="Ubuntu"
2021-10-28T20:18:41.974 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2021-10-28T20:18:41.974 INFO:teuthology.orchestra.run.smithi170.stdout:ID=ubuntu
2021-10-28T20:18:41.975 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE=debian
2021-10-28T20:18:41.975 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2021-10-28T20:18:41.975 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="20.04"
2021-10-28T20:18:41.975 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://www.ubuntu.com/"
2021-10-28T20:18:41.976 INFO:teuthology.orchestra.run.smithi170.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-10-28T20:18:41.976 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-10-28T20:18:41.976 INFO:teuthology.orchestra.run.smithi170.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-10-28T20:18:41.976 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_CODENAME=focal
2021-10-28T20:18:41.976 INFO:teuthology.orchestra.run.smithi170.stdout:UBUNTU_CODENAME=focal
2021-10-28T20:18:41.977 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2021-10-28T20:18:42.074 INFO:teuthology.dispatcher.supervisor:Running job 6465311
2021-10-28T20:18:42.075 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2021-10-28_20:09:21-rados:cephadm-wip-adk-testing-2021-10-28-1305-distro-basic-smithi/6465311 --name adking-2021-10-28_20:09:21-rados:cephadm-wip-adk-testing-2021-10-28-1305-distro-basic-smithi --description rados:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-start-upgrade 4-wait 5-upgrade-ls agent/on mon_election/classic} -- /home/teuthworker/archive/adking-2021-10-28_20:09:21-rados:cephadm-wip-adk-testing-2021-10-28-1305-distro-basic-smithi/6465311/orig.config.yaml
2021-10-28T20:18:42.079 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/adking-2021-10-28_20:09:21-rados:cephadm-wip-adk-testing-2021-10-28-1305-distro-basic-smithi/6465311
2021-10-28T20:18:42.079 INFO:teuthology.dispatcher.supervisor:Job PID: 17480
2021-10-28T20:18:42.080 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-10-29T08:19:35.920 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-10-29T08:19:36.095 INFO:teuthology.kill:Killing Pids: {17480}
2021-10-29T08:19:37.347 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2021-10-29T08:19:37.348 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2021-10-29T08:19:37.348 INFO:teuthology.misc:Compressing logs...
2021-10-29T08:19:37.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com'
2021-10-29T08:19:37.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2021-10-29T08:19:37.583 INFO:teuthology.orchestra.run.smithi164.stderr:find: '/var/lib/ceph/crash': No such file or directory
2021-10-29T08:19:37.757 INFO:teuthology.orchestra.run.smithi170.stderr:find: '/var/lib/ceph/crash': No such file or directory
2021-10-29T08:19:37.785 INFO:teuthology.orchestra.run.smithi164.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-10-29T08:19:37.785 INFO:teuthology.orchestra.run.smithi164.stderr:tar: Error is not recoverable: exiting now
2021-10-29T08:19:37.808 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-10-29T08:19:37.809 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now
2021-10-29T08:19:37.810 INFO:teuthology.misc:Compressing logs...
2021-10-29T08:19:38.607 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-10-29T08:19:38.687 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-10-29T08:19:39.222 INFO:teuthology.misc:Compressing logs...
2021-10-29T08:19:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-10-29T08:19:59.370 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2021-10-29T08:20:09.905 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2021-10-29T08:20:24.003 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-10-29T08:20:35.799 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-10-29T08:20:46.375 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2021-10-29T08:20:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2021-10-29T08:20:54.591 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2021-10-29T08:21:19.281 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-10-29T08:21:26.515 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/918d3fee-382d-11ec-8c28-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2021-10-29T08:22:16.354 INFO:teuthology.kill:No teuthology processes running
2021-10-29T08:22:17.453 INFO:teuthology.kill:Nuking machines: ['smithi164', 'smithi170']
2021-10-29T08:22:18.340 INFO:teuthology.kill:2021-10-29 08:22:18,339.339 INFO:teuthology.nuke:targets:
2021-10-29T08:22:18.341 INFO:teuthology.kill:  smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyFZ9A3yTqaR4XHBQa+gethuO82m48OmyCepTPLbDv1CJBzD9fd8rOsbeZcuWbK8jYOmMVNn6AQkMFSL1TYowmUvduCi2Co4Cyzg+Lk+JlCIxPRVOg1OAfJuaYdi5iq6kZ+hOVvX1zH6yIldrEHozzYGHRb6Q7Qnohd078BpwAXJ9UrVgmdpUJ/lvQIL9JS0f4J7dSKlUhpSBVGcr4x3t3Y857QWI1ntWVPwG1uR9o/RYrhxW0kWLvZcAZeEkHyvGyLAX5MpZ9O3ERfFOkKm5A7toWfJfw5fm3GdfE0jtNg6Q8V5yF9HuwEyRKTBatuUq4SFJY3Rj9hpxQTETNB6zt4XSYhRT2UjnCZVS1hJHJGK4wz40vpLYxtHveq/RyLrtKhVprI5T4xLJD94HrG50Vopu7lvpUxZuzuC3rXnFuRGntyRdCxKamhRDarmmzo3qjxIvZ1Fhm2FDO+cMEA8aotiBuLXsN+sTvCYMHkZFWTdogI8FHNBLhAXdMYoJor00=
2021-10-29T08:22:18.341 INFO:teuthology.kill:  smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0CeuwHtDhExbsYQITKmrKwg5DrwGfLL89Azwd5zOoqVkyndUmsNzEKSrsbXzNlAGKsaTYiYx+qBfEGDKQwlvBlZfqwNHE70eNsai5VCFbkdNCpiqqFzKJLB9z+FOndBTjk8eaUsUoskSsiBb0oUYZwdzeDASfFg11D8ViX7k3v6obJ0E3EoS7KJqZnwBKCGjQ3NbtGWtO1LSPec3/TOR5xrtVt7j+rPhTiLYioqCq+vZLglVNI1IWOD5zvHhVCPbq30Cypa1nNBr8b1wlFpEUsFONhC4U2F45Ys7fHp2IjGOCSDW+MBGHWFLpR+ZWxj9V5Xls9fFmvWEN0AVLQaJqQdxTjgS2LcHdEbS8higbfnvcbmBSPy1E7ed1CK34w1qpPp7hXRpDLOBkcbi7l6bLbbzDaZfvcHNp7IBJ0AHZl5jRGdVRMckXjhdDuhNOH+Fih+FeETvQSyGX2FrdxWhoXEEBhu0Ufmol2zURCQ4qDkO//Om1GxkKuLnnqEHWUCk=
2021-10-29T08:22:18.366 INFO:teuthology.kill:2021-10-29 08:22:18,366.366 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-10-29T08:22:18.369 INFO:teuthology.kill:2021-10-29 08:22:18,369.369 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-10-29T08:22:18.444 INFO:teuthology.kill:2021-10-29 08:22:18,443.443 INFO:teuthology.orchestra.console:Power off smithi164
2021-10-29T08:22:18.455 INFO:teuthology.kill:2021-10-29 08:22:18,455.455 INFO:teuthology.orchestra.console:Power off smithi170
2021-10-29T08:22:30.922 INFO:teuthology.kill:2021-10-29 08:22:30,922.922 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2021-10-29T08:22:30.927 INFO:teuthology.kill:2021-10-29 08:22:30,927.927 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2021-10-29T08:22:31.075 INFO:teuthology.kill:2021-10-29 08:22:31,074.074 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com
2021-10-29T08:22:31.079 INFO:teuthology.kill:2021-10-29 08:22:31,079.079 INFO:teuthology.lock.ops:unlocked: smithi164.front.sepia.ceph.com
2021-10-29T08:24:31.375 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-10-29T08:24:31.464 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi164 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-10-28_23:09:03-rados-wip-yuri2-testing-2021-10-28-1343-distro-basic-smithi/6466002