2021-10-28T21:34:11.235 INFO:root:teuthology version: 1.1.0-c56135d1
2021-10-28T21:34:11.360 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi110.front.sepia.ceph.com' reimaging
2021-10-28T21:34:11.361 INFO:teuthology.lock.ops:Updating [ubuntu@smithi110.front.sepia.ceph.com]: reset os type and version on server
2021-10-28T21:34:11.362 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2021-10-28T21:34:11.379 INFO:teuthology.lock.ops:Node 'ubuntu@smithi110.front.sepia.ceph.com' reimaging is complete
2021-10-28T21:34:11.379 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi171.front.sepia.ceph.com' reimaging
2021-10-28T21:34:11.380 INFO:teuthology.lock.ops:Updating [ubuntu@smithi171.front.sepia.ceph.com]: reset os type and version on server
2021-10-28T21:34:11.380 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server
2021-10-28T21:34:11.398 INFO:teuthology.lock.ops:Node 'ubuntu@smithi171.front.sepia.ceph.com' reimaging is complete
2021-10-28T21:34:11.610 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of ubuntu 20.04
2021-10-28T21:34:11.651 INFO:teuthology.provision.fog.smithi171:Scheduling deploy of ubuntu 20.04
2021-10-28T21:34:12.180 INFO:teuthology.orchestra.console:Power off smithi110
2021-10-28T21:34:12.230 INFO:teuthology.orchestra.console:Power off smithi171
2021-10-28T21:34:12.339 INFO:teuthology.orchestra.console:Power off for smithi110 completed
2021-10-28T21:34:12.441 INFO:teuthology.orchestra.console:Power on smithi110
2021-10-28T21:34:16.692 INFO:teuthology.orchestra.console:Power on for smithi110 completed
2021-10-28T21:34:16.793 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish
2021-10-28T21:34:24.712 INFO:teuthology.orchestra.console:Power off for smithi171 completed
2021-10-28T21:34:24.813 INFO:teuthology.orchestra.console:Power on smithi171
2021-10-28T21:34:29.068 INFO:teuthology.orchestra.console:Power on for smithi171 completed
2021-10-28T21:34:29.169 INFO:teuthology.provision.fog.smithi171:Waiting for deploy to finish
2021-10-28T21:38:03.320 ERROR:teuthology.orchestra.connection:Error authenticating with smithi171.front.sepia.ceph.com: Authentication failed.
2021-10-28T21:40:19.614 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-10-28T21:40:20.148 INFO:teuthology.provision.fog.smithi171:Node is ready
2021-10-28T21:40:20.155 INFO:teuthology.orchestra.run.smithi171.stdout:smithi171.front.sepia.ceph.com
2021-10-28T21:40:20.161 INFO:teuthology.orchestra.run.smithi171.stdout:172.21.15.171 smithi171.front.sepia.ceph.com smithi171
2021-10-28T21:40:20.376 INFO:teuthology.provision.fog.smithi171:Deploy complete!
2021-10-28T21:40:36.885 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-10-28T21:40:37.504 INFO:teuthology.provision.fog.smithi110:Node is ready
2021-10-28T21:40:37.512 INFO:teuthology.orchestra.run.smithi110.stdout:smithi110.front.sepia.ceph.com
2021-10-28T21:40:37.562 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.15.110 smithi110.front.sepia.ceph.com smithi110
2021-10-28T21:40:37.768 INFO:teuthology.provision.fog.smithi110:Deploy complete!
2021-10-28T21:40:37.993 INFO:teuthology.lock.ops:Checking smithi110.front.sepia.ceph.com
2021-10-28T21:40:37.994 INFO:teuthology.lock.ops:New key found. Updating...
2021-10-28T21:40:38.022 INFO:teuthology.lock.ops:Checking smithi171.front.sepia.ceph.com
2021-10-28T21:40:38.023 INFO:teuthology.lock.ops:New key found. Updating...
2021-10-28T21:40:38.044 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: set os type and version on server
2021-10-28T21:40:38.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com'
2021-10-28T21:40:38.235 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64
2021-10-28T21:40:38.242 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu"
2021-10-28T21:40:38.243 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2021-10-28T21:40:38.243 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu
2021-10-28T21:40:38.243 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian
2021-10-28T21:40:38.244 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2021-10-28T21:40:38.244 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="20.04"
2021-10-28T21:40:38.245 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/"
2021-10-28T21:40:38.245 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-10-28T21:40:38.245 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-10-28T21:40:38.245 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-10-28T21:40:38.245 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=focal
2021-10-28T21:40:38.246 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=focal
2021-10-28T21:40:38.247 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2021-10-28T21:40:38.267 INFO:teuthology.lock.ops:Updating [smithi171.front.sepia.ceph.com]: set os type and version on server
2021-10-28T21:40:38.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com'
2021-10-28T21:40:38.455 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64
2021-10-28T21:40:38.462 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="Ubuntu"
2021-10-28T21:40:38.462 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2021-10-28T21:40:38.463 INFO:teuthology.orchestra.run.smithi171.stdout:ID=ubuntu
2021-10-28T21:40:38.463 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE=debian
2021-10-28T21:40:38.464 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2021-10-28T21:40:38.464 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="20.04"
2021-10-28T21:40:38.464 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://www.ubuntu.com/"
2021-10-28T21:40:38.465 INFO:teuthology.orchestra.run.smithi171.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-10-28T21:40:38.465 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-10-28T21:40:38.465 INFO:teuthology.orchestra.run.smithi171.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-10-28T21:40:38.465 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_CODENAME=focal
2021-10-28T21:40:38.466 INFO:teuthology.orchestra.run.smithi171.stdout:UBUNTU_CODENAME=focal
2021-10-28T21:40:38.466 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server
2021-10-28T21:40:38.565 INFO:teuthology.dispatcher.supervisor:Running job 6465404
2021-10-28T21:40:38.566 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2021-10-28_21:32:34-rados-wip-43687-distro-basic-smithi/6465404 --name nojha-2021-10-28_21:32:34-rados-wip-43687-distro-basic-smithi --description rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-start-upgrade 4-wait 5-upgrade-ls mon_election/classic} -- /home/teuthworker/archive/nojha-2021-10-28_21:32:34-rados-wip-43687-distro-basic-smithi/6465404/orig.config.yaml
2021-10-28T21:40:38.570 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2021-10-28_21:32:34-rados-wip-43687-distro-basic-smithi/6465404
2021-10-28T21:40:38.571 INFO:teuthology.dispatcher.supervisor:Job PID: 22423
2021-10-28T21:40:38.571 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-10-29T09:40:58.639 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-10-29T09:40:58.808 INFO:teuthology.kill:Killing Pids: {22423}
2021-10-29T09:40:59.954 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2021-10-29T09:40:59.955 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2021-10-29T09:40:59.955 INFO:teuthology.misc:Compressing logs...
2021-10-29T09:40:59.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com'
2021-10-29T09:41:00.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com'
2021-10-29T09:41:00.143 INFO:teuthology.orchestra.run.smithi110.stderr:find: '/var/lib/ceph/crash': No such file or directory
2021-10-29T09:41:00.315 INFO:teuthology.orchestra.run.smithi171.stderr:find: '/var/lib/ceph/crash': No such file or directory
2021-10-29T09:41:00.347 INFO:teuthology.orchestra.run.smithi110.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-10-29T09:41:00.347 INFO:teuthology.orchestra.run.smithi110.stderr:tar: Error is not recoverable: exiting now
2021-10-29T09:41:00.365 INFO:teuthology.orchestra.run.smithi171.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-10-29T09:41:00.365 INFO:teuthology.orchestra.run.smithi171.stderr:tar: Error is not recoverable: exiting now
2021-10-29T09:41:00.368 INFO:teuthology.misc:Compressing logs...
2021-10-29T09:41:01.208 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-10-29T09:41:01.232 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-10-29T09:41:01.757 INFO:teuthology.misc:Compressing logs...
2021-10-29T09:41:19.276 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-10-29T09:41:22.144 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2021-10-29T09:41:30.131 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2021-10-29T09:41:46.805 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-10-29T09:41:55.151 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-10-29T09:42:09.290 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2021-10-29T09:42:10.010 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2021-10-29T09:42:17.262 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2021-10-29T09:42:41.426 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2021-10-29T09:42:42.145 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/e75dd7f2-3838-11ec-8c28-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-10-29T09:43:32.595 INFO:teuthology.kill:No teuthology processes running
2021-10-29T09:43:33.792 INFO:teuthology.kill:Nuking machines: ['smithi110', 'smithi171']
2021-10-29T09:43:34.737 INFO:teuthology.kill:2021-10-29 09:43:34,736.736 INFO:teuthology.nuke:targets:
2021-10-29T09:43:34.738 INFO:teuthology.kill:  smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRjgJOITthIt9yfl4JYYyaa2oMn/ofhSoi8z9qWYL6umQLBvX+AteBPaOnwumKznHbQtotLtec+tTNoMPKKEuIdyZc/gj+mrIbEgNwR37pQmNzMvTDbU4snRmqmcQsMI6wD7QhTQ22ESTp6FCEsQ/sUyJoGHlMFEHq/oJ863l/3Loamp99WV4Hk7uyLWytsIkME7msao3tB5NMyj1IyKx6ZIAJK1M2Q9TLPkJPlScxaojIQsE+FJpWA7fMXQqz/VO6z7zK6b61BkVQt3TfQoNI1/BKFaYFj1RCbD/zQe6DkDAGbAJqtlIGzLKPWLzLS9bwlui7LhdBmWMJZJXi7Te+j2JvE/cLMSOGuVBp5ulsReBf4wZ9yF5mRQrjvzEp19CYCs7C7xUle5BAgk9jdTsnv++roXrlq/GJoArLpKbGqi8hVLvhsdu8XCBGZZs+JEm9UUT5c978rWJoieuECEMWlDbJRfjgAzKk9lugjH3Zn00aElYfXcXC1dGeq+Uxfrk=
2021-10-29T09:43:34.738 INFO:teuthology.kill:  smithi171.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6YMV23PlLNGrOlu+3M4g+4mBOm3/n/+/XvtZpuNuemhnNTaOUKhGSVrEclXWctZRqUVQr/m+7E7skL2aC1Jl3EIeQTynbt86VOQOsIqhMXT/7rtrR1YSVDPQjmZPhTeX3QWSeXIXwtIcNIWpZpzIAFiMJwizC6BKekk3CPb4V2xX9gX2v40EB9PFQl9DVsE5nGOM/qpdIZUjgEaDAz4jsZRsj5ZhBTvR/e5JVwyr7k1yerlmzVJhIwC2Up7CHPoZIGP3Y3omwm+MzAbK70Kq9JAzllXBmrC+3pPTcSm8CFBeRsCHRvGyfRcur2UuN+tXn7aTYZxI/89k1xj2h6xgelGkfzRAZCfhX5jTEGZukm5BBi40b/giA7F8awB0mpJZkt/gcSOTEi6+tdqYVcB4HsFQcVRQVVoKVGJfi3hRQXyUA6BrXf6PUN7E0miC9rlkZOhBP6jp+kZMLl3Rlzvbc/ytNOgm/h+7ui6927w/g/q6q7jKnmMbC/v4GUJEa4C8=
2021-10-29T09:43:34.764 INFO:teuthology.kill:2021-10-29 09:43:34,764.764 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-10-29T09:43:34.766 INFO:teuthology.kill:2021-10-29 09:43:34,766.766 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-10-29T09:43:34.844 INFO:teuthology.kill:2021-10-29 09:43:34,844.844 INFO:teuthology.orchestra.console:Power off smithi110
2021-10-29T09:43:34.859 INFO:teuthology.kill:2021-10-29 09:43:34,859.859 INFO:teuthology.orchestra.console:Power off smithi171
2021-10-29T09:43:43.220 INFO:teuthology.kill:2021-10-29 09:43:43,219.219 INFO:teuthology.orchestra.console:Power off for smithi171 completed
2021-10-29T09:43:43.367 INFO:teuthology.kill:2021-10-29 09:43:43,366.366 INFO:teuthology.lock.ops:unlocked: smithi171.front.sepia.ceph.com
2021-10-29T09:43:47.323 INFO:teuthology.kill:2021-10-29 09:43:47,323.323 INFO:teuthology.orchestra.console:Power off for smithi110 completed
2021-10-29T09:43:47.470 INFO:teuthology.kill:2021-10-29 09:43:47,470.470 INFO:teuthology.lock.ops:unlocked: smithi110.front.sepia.ceph.com
2021-10-29T09:45:47.669 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-10-29T09:45:47.756 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi110 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-10-28_23:09:03-rados-wip-yuri2-testing-2021-10-28-1343-distro-basic-smithi/6466069
2021-10-29T09:45:47.757 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi171 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-10-28_23:09:03-rados-wip-yuri2-testing-2021-10-28-1343-distro-basic-smithi/6466069