2021-11-10T18:04:10.038 INFO:root:teuthology version: 1.1.0-c56135d1
2021-11-10T18:04:10.155 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi027.front.sepia.ceph.com' reimaging
2021-11-10T18:04:10.156 INFO:teuthology.lock.ops:Updating [ubuntu@smithi027.front.sepia.ceph.com]: reset os type and version on server
2021-11-10T18:04:10.156 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2021-11-10T18:04:10.175 INFO:teuthology.lock.ops:Node 'ubuntu@smithi027.front.sepia.ceph.com' reimaging is complete
2021-11-10T18:04:10.175 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi104.front.sepia.ceph.com' reimaging
2021-11-10T18:04:10.175 INFO:teuthology.lock.ops:Updating [ubuntu@smithi104.front.sepia.ceph.com]: reset os type and version on server
2021-11-10T18:04:10.176 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2021-11-10T18:04:10.193 INFO:teuthology.lock.ops:Node 'ubuntu@smithi104.front.sepia.ceph.com' reimaging is complete
2021-11-10T18:04:10.381 INFO:teuthology.provision.fog.smithi027:Scheduling deploy of ubuntu 20.04
2021-11-10T18:04:10.391 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of ubuntu 20.04
2021-11-10T18:04:10.722 INFO:teuthology.orchestra.console:Power off smithi104
2021-11-10T18:04:10.737 INFO:teuthology.orchestra.console:Power off smithi027
2021-11-10T18:04:19.100 INFO:teuthology.orchestra.console:Power off for smithi027 completed
2021-11-10T18:04:19.201 INFO:teuthology.orchestra.console:Power on smithi027
2021-11-10T18:04:23.207 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2021-11-10T18:04:23.308 INFO:teuthology.orchestra.console:Power on smithi104
2021-11-10T18:04:27.561 INFO:teuthology.orchestra.console:Power on for smithi027 completed
2021-11-10T18:04:27.565 INFO:teuthology.orchestra.console:Power on for smithi104 completed
2021-11-10T18:04:27.663 INFO:teuthology.provision.fog.smithi027:Waiting for deploy to finish
2021-11-10T18:04:27.667 INFO:teuthology.provision.fog.smithi104:Waiting for deploy to finish
2021-11-10T18:09:32.112 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-10T18:09:32.702 INFO:teuthology.provision.fog.smithi027:Node is ready
2021-11-10T18:09:32.708 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027.front.sepia.ceph.com
2021-11-10T18:09:32.716 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27 smithi027.front.sepia.ceph.com smithi027
2021-11-10T18:09:32.921 INFO:teuthology.provision.fog.smithi027:Deploy complete!
2021-11-10T18:09:47.251 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-10T18:09:47.748 INFO:teuthology.provision.fog.smithi104:Node is ready
2021-11-10T18:09:47.754 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104.front.sepia.ceph.com
2021-11-10T18:09:47.804 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104 smithi104.front.sepia.ceph.com smithi104
2021-11-10T18:09:48.043 INFO:teuthology.provision.fog.smithi104:Deploy complete!
2021-11-10T18:09:48.222 INFO:teuthology.lock.ops:Checking smithi027.front.sepia.ceph.com
2021-11-10T18:09:48.223 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-10T18:09:48.243 INFO:teuthology.lock.ops:Checking smithi104.front.sepia.ceph.com
2021-11-10T18:09:48.244 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-10T18:09:48.260 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: set os type and version on server
2021-11-10T18:09:48.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2021-11-10T18:09:48.530 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64
2021-11-10T18:09:48.537 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="Ubuntu"
2021-11-10T18:09:48.537 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2021-11-10T18:09:48.537 INFO:teuthology.orchestra.run.smithi027.stdout:ID=ubuntu
2021-11-10T18:09:48.537 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE=debian
2021-11-10T18:09:48.538 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2021-11-10T18:09:48.538 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="20.04"
2021-11-10T18:09:48.538 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://www.ubuntu.com/"
2021-11-10T18:09:48.538 INFO:teuthology.orchestra.run.smithi027.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-11-10T18:09:48.539 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-11-10T18:09:48.539 INFO:teuthology.orchestra.run.smithi027.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-11-10T18:09:48.539 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_CODENAME=focal
2021-11-10T18:09:48.539 INFO:teuthology.orchestra.run.smithi027.stdout:UBUNTU_CODENAME=focal
2021-11-10T18:09:48.540 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2021-11-10T18:09:48.558 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: set os type and version on server
2021-11-10T18:09:48.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com'
2021-11-10T18:09:48.734 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64
2021-11-10T18:09:48.740 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Ubuntu"
2021-11-10T18:09:48.740 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2021-11-10T18:09:48.741 INFO:teuthology.orchestra.run.smithi104.stdout:ID=ubuntu
2021-11-10T18:09:48.741 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE=debian
2021-11-10T18:09:48.741 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2021-11-10T18:09:48.741 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="20.04"
2021-11-10T18:09:48.741 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.ubuntu.com/"
2021-11-10T18:09:48.742 INFO:teuthology.orchestra.run.smithi104.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-11-10T18:09:48.742 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-11-10T18:09:48.742 INFO:teuthology.orchestra.run.smithi104.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-11-10T18:09:48.742 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_CODENAME=focal
2021-11-10T18:09:48.742 INFO:teuthology.orchestra.run.smithi104.stdout:UBUNTU_CODENAME=focal
2021-11-10T18:09:48.743 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2021-11-10T18:09:48.841 INFO:teuthology.dispatcher.supervisor:Running job 6484843
2021-11-10T18:09:48.841 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_akupczyk@teuthology --archive /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484843 --name akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi --description rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-start-upgrade 4-wait 5-upgrade-ls mon_election/classic} -- /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484843/orig.config.yaml
2021-11-10T18:09:48.845 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484843
2021-11-10T18:09:48.845 INFO:teuthology.dispatcher.supervisor:Job PID: 13358
2021-11-10T18:09:48.846 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-11T06:10:41.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-11T06:10:42.145 INFO:teuthology.kill:Killing Pids: {13358}
2021-11-11T06:10:43.310 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2021-11-11T06:10:43.311 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a']
2021-11-11T06:10:43.312 INFO:teuthology.misc:Compressing logs...
2021-11-11T06:10:43.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2021-11-11T06:10:43.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com'
2021-11-11T06:10:43.501 INFO:teuthology.orchestra.run.smithi027.stderr:find: '/var/lib/ceph/crash': No such file or directory
2021-11-11T06:10:43.676 INFO:teuthology.orchestra.run.smithi104.stderr:find: '/var/lib/ceph/crash': No such file or directory
2021-11-11T06:10:43.704 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-11-11T06:10:43.705 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now
2021-11-11T06:10:43.723 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-11-11T06:10:43.724 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now
2021-11-11T06:10:43.726 INFO:teuthology.misc:Compressing logs...
2021-11-11T06:10:44.510 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-11T06:10:44.635 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-11T06:10:45.164 INFO:teuthology.misc:Compressing logs...
2021-11-11T06:11:07.465 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2021-11-11T06:11:09.538 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-11-11T06:11:20.674 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2021-11-11T06:11:34.073 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-11-11T06:11:52.485 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-11-11T06:11:58.985 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2021-11-11T06:12:06.651 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2021-11-11T06:12:08.135 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2021-11-11T06:12:33.911 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-11-11T06:12:43.974 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/7e5e0186-4252-11ec-8c2c-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2021-11-11T06:13:40.708 INFO:teuthology.kill:No teuthology processes running
2021-11-11T06:13:41.947 INFO:teuthology.kill:Nuking machines: ['smithi027', 'smithi104']
2021-11-11T06:13:42.943 INFO:teuthology.kill:2021-11-11 06:13:42,943.943 INFO:teuthology.nuke:targets:
2021-11-11T06:13:42.944 INFO:teuthology.kill:  smithi027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDATnCe/3QOuLdY/+6PWsVUSIUyu+BHgmfxW4aF9z1YkzidoH6vAWRfOQOIsnwXo7FmXfwJ5k2Ed571lxnW6ZhhK1igN1QsSsz/JOQHptIhSlgGaEcI7Xl6DKsUxkOLMpvWYzqQ9Z70cqZOZT961Sm1H4Obhl3MKT3c69guLCTkVU4XTKHJjUMHdB12Q8k5+dvtQbirnTJxPRQ9inPY9IkyopUPqeXoESQmW8S596rwi+DWdLorbgquQMceQLzC5isFfhJzFKzY4qpbK5babvviiGxHYjgU795HvnbMuRK6PoHrl3AQrKIsgxGNZRgcXIH9L5V2eqcX7L1RpZ6/B4wcd3/OFZCsPOQO5/1tjAzY29QS4/lzEmIRZhJaB1A1B8lZgIblUj+MmBjvQc4NvMLXA6xdMuZA/qejTVJVk9iBz/TQWpgaZEStn/28umZH6XDVu2ioKBa6I2lTCKVuGpVD2rHYqmpWDJdnipYvoKO2xRns2my1x8efQJU6VcN7Dd8=
2021-11-11T06:13:42.945 INFO:teuthology.kill:  smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCePd+81KYsvfQoMfsco1Z/7JrlAoXJVLtLvEVMNMW48+yilctlubK8KsfVsZCj2HkvJ1kysreVrQTWV1zxWhb3LEExyOt4GuX11zvfNnTnlt2vdRGwJPfmRFVm81jq/65r8Awj+Zq3zAwUiKuDsePkDmO/uGxSuaNyJLYvVlaFPoK2nyDMqZYtQLfuHyqNaHzh8bLI9TYwifOG5ggzObbcxgN0sHpROYe+t8XPIIQaYtgXruwGKIvao2F8uIjE6lHHdlewfbfBieIntlvDLUZlHQ0muROdcv8xAnYdI+V76zVIlueVDD29E9mkluRldahTQ1QdX0q7uEcg2YbD8EMlCuXWRcTo5C/tALTq/krbXRxduJEwRnGZjH1JB4iZJwld+HTDrch0kZZ+L3pn96+ZkP2lxH5GPVtOLrQ72FB+UeylHEi3bZzu33xp3ZLMEWgR4fiC1GZTQT0PBZXdMcxh7vTgFsak8Z2b5X0mbhbe+XVISNW55xf3JF2yXnF+SLs=
2021-11-11T06:13:42.970 INFO:teuthology.kill:2021-11-11 06:13:42,969.969 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T06:13:42.973 INFO:teuthology.kill:2021-11-11 06:13:42,973.973 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T06:13:43.053 INFO:teuthology.kill:2021-11-11 06:13:43,052.052 INFO:teuthology.orchestra.console:Power off smithi027
2021-11-11T06:13:43.065 INFO:teuthology.kill:2021-11-11 06:13:43,065.065 INFO:teuthology.orchestra.console:Power off smithi104
2021-11-11T06:13:51.420 INFO:teuthology.kill:2021-11-11 06:13:51,420.420 INFO:teuthology.orchestra.console:Power off for smithi027 completed
2021-11-11T06:13:51.571 INFO:teuthology.kill:2021-11-11 06:13:51,570.570 INFO:teuthology.lock.ops:unlocked: smithi027.front.sepia.ceph.com
2021-11-11T06:13:55.534 INFO:teuthology.kill:2021-11-11 06:13:55,534.534 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2021-11-11T06:13:55.682 INFO:teuthology.kill:2021-11-11 06:13:55,682.682 INFO:teuthology.lock.ops:unlocked: smithi104.front.sepia.ceph.com
2021-11-11T06:15:56.001 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-11T06:15:56.086 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi027 but it was locked by another job: /home/teuthworker/archive/benhanokh-2021-11-10_07:27:45-rados-restore_alloc_file-distro-basic-smithi/6495342
2021-11-11T06:15:56.087 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi104 but it was locked by another job: /home/teuthworker/archive/benhanokh-2021-11-10_07:27:45-rados-restore_alloc_file-distro-basic-smithi/6495342