2021-11-11T00:10:56.016 INFO:root:teuthology version: 1.1.0-c56135d1
2021-11-11T00:10:56.150 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi043.front.sepia.ceph.com' reimaging
2021-11-11T00:10:56.151 INFO:teuthology.lock.ops:Updating [ubuntu@smithi043.front.sepia.ceph.com]: reset os type and version on server
2021-11-11T00:10:56.151 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server
2021-11-11T00:10:56.170 INFO:teuthology.lock.ops:Node 'ubuntu@smithi043.front.sepia.ceph.com' reimaging is complete
2021-11-11T00:10:56.171 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi092.front.sepia.ceph.com' reimaging
2021-11-11T00:10:56.171 INFO:teuthology.lock.ops:Updating [ubuntu@smithi092.front.sepia.ceph.com]: reset os type and version on server
2021-11-11T00:10:56.171 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server
2021-11-11T00:10:56.192 INFO:teuthology.lock.ops:Node 'ubuntu@smithi092.front.sepia.ceph.com' reimaging is complete
2021-11-11T00:10:56.379 INFO:teuthology.provision.fog.smithi043:Scheduling deploy of ubuntu 20.04
2021-11-11T00:10:56.419 INFO:teuthology.provision.fog.smithi092:Scheduling deploy of ubuntu 20.04
2021-11-11T00:10:56.736 INFO:teuthology.orchestra.console:Power off smithi043
2021-11-11T00:10:56.777 INFO:teuthology.orchestra.console:Power off smithi092
2021-11-11T00:11:05.113 INFO:teuthology.orchestra.console:Power off for smithi043 completed
2021-11-11T00:11:05.142 INFO:teuthology.orchestra.console:Power off for smithi092 completed
2021-11-11T00:11:05.215 INFO:teuthology.orchestra.console:Power on smithi043
2021-11-11T00:11:05.245 INFO:teuthology.orchestra.console:Power on smithi092
2021-11-11T00:11:13.588 INFO:teuthology.orchestra.console:Power on for smithi043 completed
2021-11-11T00:11:13.609 INFO:teuthology.orchestra.console:Power on for smithi092 completed
2021-11-11T00:11:13.690 INFO:teuthology.provision.fog.smithi043:Waiting for deploy to finish
2021-11-11T00:11:13.710 INFO:teuthology.provision.fog.smithi092:Waiting for deploy to finish
2021-11-11T00:16:18.305 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-11T00:16:18.838 INFO:teuthology.provision.fog.smithi043:Node is ready
2021-11-11T00:16:18.845 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043.front.sepia.ceph.com
2021-11-11T00:16:18.852 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43 smithi043.front.sepia.ceph.com smithi043
2021-11-11T00:16:19.063 INFO:teuthology.provision.fog.smithi043:Deploy complete!
2021-11-11T00:16:33.401 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-11T00:16:33.970 INFO:teuthology.provision.fog.smithi092:Node is ready
2021-11-11T00:16:33.978 INFO:teuthology.orchestra.run.smithi092.stdout:smithi092.front.sepia.ceph.com
2021-11-11T00:16:33.986 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.92 smithi092.front.sepia.ceph.com smithi092
2021-11-11T00:16:34.231 INFO:teuthology.provision.fog.smithi092:Deploy complete!
2021-11-11T00:16:34.414 INFO:teuthology.lock.ops:Checking smithi043.front.sepia.ceph.com
2021-11-11T00:16:34.415 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-11T00:16:34.435 INFO:teuthology.lock.ops:Checking smithi092.front.sepia.ceph.com
2021-11-11T00:16:34.435 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-11T00:16:34.453 INFO:teuthology.lock.ops:Updating [smithi043.front.sepia.ceph.com]: set os type and version on server
2021-11-11T00:16:34.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com'
2021-11-11T00:16:34.717 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64
2021-11-11T00:16:34.766 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="Ubuntu"
2021-11-11T00:16:34.766 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2021-11-11T00:16:34.766 INFO:teuthology.orchestra.run.smithi043.stdout:ID=ubuntu
2021-11-11T00:16:34.767 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE=debian
2021-11-11T00:16:34.767 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2021-11-11T00:16:34.767 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="20.04"
2021-11-11T00:16:34.767 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://www.ubuntu.com/"
2021-11-11T00:16:34.768 INFO:teuthology.orchestra.run.smithi043.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-11-11T00:16:34.768 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-11-11T00:16:34.768 INFO:teuthology.orchestra.run.smithi043.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-11-11T00:16:34.768 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_CODENAME=focal
2021-11-11T00:16:34.768 INFO:teuthology.orchestra.run.smithi043.stdout:UBUNTU_CODENAME=focal
2021-11-11T00:16:34.770 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server
2021-11-11T00:16:34.788 INFO:teuthology.lock.ops:Updating [smithi092.front.sepia.ceph.com]: set os type and version on server
2021-11-11T00:16:34.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com'
2021-11-11T00:16:34.958 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64
2021-11-11T00:16:34.964 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Ubuntu"
2021-11-11T00:16:34.964 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2021-11-11T00:16:34.964 INFO:teuthology.orchestra.run.smithi092.stdout:ID=ubuntu
2021-11-11T00:16:34.964 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE=debian
2021-11-11T00:16:34.965 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2021-11-11T00:16:34.965 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="20.04"
2021-11-11T00:16:34.965 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.ubuntu.com/"
2021-11-11T00:16:34.965 INFO:teuthology.orchestra.run.smithi092.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-11-11T00:16:34.966 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-11-11T00:16:34.966 INFO:teuthology.orchestra.run.smithi092.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-11-11T00:16:34.966 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_CODENAME=focal
2021-11-11T00:16:34.966 INFO:teuthology.orchestra.run.smithi092.stdout:UBUNTU_CODENAME=focal
2021-11-11T00:16:34.967 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server
2021-11-11T00:16:35.060 INFO:teuthology.dispatcher.supervisor:Running job 6484895
2021-11-11T00:16:35.061 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_akupczyk@teuthology --archive /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484895 --name akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi --description rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-start-upgrade 4-wait 5-upgrade-ls mon_election/connectivity} -- /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484895/orig.config.yaml
2021-11-11T00:16:35.064 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484895
2021-11-11T00:16:35.065 INFO:teuthology.dispatcher.supervisor:Job PID: 20699
2021-11-11T00:16:35.065 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-11T12:17:29.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-11T12:17:29.470 INFO:teuthology.kill:Killing Pids: {20699}
2021-11-11T12:17:30.627 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2021-11-11T12:17:30.628 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2021-11-11T12:17:30.628 INFO:teuthology.misc:Compressing logs...
2021-11-11T12:17:30.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com'
2021-11-11T12:17:30.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com'
2021-11-11T12:17:30.870 INFO:teuthology.orchestra.run.smithi043.stderr:find: '/var/lib/ceph/crash': No such file or directory
2021-11-11T12:17:31.085 INFO:teuthology.orchestra.run.smithi092.stderr:find: '/var/lib/ceph/crash': No such file or directory
2021-11-11T12:17:31.182 INFO:teuthology.orchestra.run.smithi043.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-11-11T12:17:31.183 INFO:teuthology.orchestra.run.smithi043.stderr:tar: Error is not recoverable: exiting now
2021-11-11T12:17:31.201 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-11-11T12:17:31.201 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now
2021-11-11T12:17:31.203 INFO:teuthology.misc:Compressing logs...
2021-11-11T12:17:31.986 INFO:teuthology.orchestra.run.smithi092.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-11T12:17:32.110 INFO:teuthology.orchestra.run.smithi043.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-11T12:17:32.628 INFO:teuthology.misc:Compressing logs...
2021-11-11T12:17:53.754 INFO:teuthology.orchestra.run.smithi043.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-11-11T12:17:54.483 INFO:teuthology.orchestra.run.smithi092.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2021-11-11T12:18:05.907 INFO:teuthology.orchestra.run.smithi043.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2021-11-11T12:18:20.160 INFO:teuthology.orchestra.run.smithi092.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-11-11T12:18:32.824 INFO:teuthology.orchestra.run.smithi043.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-11-11T12:18:43.890 INFO:teuthology.orchestra.run.smithi092.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2021-11-11T12:18:48.397 INFO:teuthology.orchestra.run.smithi043.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2021-11-11T12:18:51.974 INFO:teuthology.orchestra.run.smithi092.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2021-11-11T12:19:17.810 INFO:teuthology.orchestra.run.smithi092.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-11-11T12:19:23.635 INFO:teuthology.orchestra.run.smithi043.stderr:gzip: /var/log/ceph/c0dac99e-4285-11ec-8c2c-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2021-11-11T12:20:17.987 INFO:teuthology.kill:No teuthology processes running
2021-11-11T12:20:19.092 INFO:teuthology.kill:Nuking machines: ['smithi043', 'smithi092']
2021-11-11T12:20:20.017 INFO:teuthology.kill:2021-11-11 12:20:20,017.017 INFO:teuthology.nuke:targets:
2021-11-11T12:20:20.018 INFO:teuthology.kill:  smithi043.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIlNY5Sv7lZDm300cjskjcpXaJbk0yw5W8HMpixSQOM/x0IguSNsMcTyEN8+mNaq5e0vGDxpJw+A+T/DTopKfP9MFcaxbK5o8cNKP+RXKfMP+0zlL0gI7xkMETa8PL6AYwDg7KYVKecal2vvm3fL16nv88f45niFC/mLMsRuIYiSZEJ3dCq5RX2MNh/XQFf5dQCwGzxJskUL3pNbi/1N88QQMtK0FrHkCvbR/z8aKPaLAKDPuPtzbF9YmyaspEUHMNPF0n3v6Gw2qAVdLIMEp3B+dtLQ39598Bkh2bE72n2ZE1kxDgSxSdFPZ85WnhZfYX0EurMzwHdIAVbTVt6w4OY3Drg7tuEBTRijHciEinnuci1JO8XP4v+6PMgur8u3LRV1egxOIqypRymjgXQkLwou7jMIrAQYKSuJb/V0Yuj+FnhZ0uc9liYsMIDR2oj3UUNKYXOdyy91n17Vmlf0L6+ywU2RRJKrOJRu9ZRyTjAsucIbdoP5i+j3NobQvo4yc=
2021-11-11T12:20:20.018 INFO:teuthology.kill:  smithi092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvInq5Zv/cpAy6xJNHDH96fJlmPwYqna707r8tmJY4x5LfYoaXJw5gKqDunpFHkPSSB6G21mQO/D6obNnmtTZN4G/Q8qQ05jXULkCIfgI4o2AiY1E3yCPmVQe+IOtCdrzYonl5eVNU4x8oBFXHPPDamg7mxMuY+R1MBXzJDBmmH5pLv5JPPhz0fobCSTYOAzCBvItOTVqIbx/5qlDwGLHgieVvyUbhMvsuZ8Gx4GlefOh7TnDs1O7htTj4o3diFuU9Ffl2QshldDHMKPnV7REIVq5ghgAd7Ji9lHrNVFvICckFj/59nwFwgW6gRFa4RdSDhmv9G9pS73r/4i5/3EO3xZnbTZmPfDfrGPd5Mf772fXeS34bMNOEiSqDlkRSqAjl+hGMOP58FRKiKqkSo/EmzfVAmYmHeWijccEt+TpO2sZu6S3ellUnIE4KFZWcgh9eZOR1kHFGtfn28eu5bNYgEQEF9nXvA6ybnwhuZcrNxEGmxrmdl0XVOj6zTv0JI4E=
2021-11-11T12:20:20.043 INFO:teuthology.kill:2021-11-11 12:20:20,043.043 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T12:20:20.046 INFO:teuthology.kill:2021-11-11 12:20:20,046.046 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T12:20:20.119 INFO:teuthology.kill:2021-11-11 12:20:20,119.119 INFO:teuthology.orchestra.console:Power off smithi092
2021-11-11T12:20:20.132 INFO:teuthology.kill:2021-11-11 12:20:20,132.132 INFO:teuthology.orchestra.console:Power off smithi043
2021-11-11T12:20:28.492 INFO:teuthology.kill:2021-11-11 12:20:28,492.492 INFO:teuthology.orchestra.console:Power off for smithi043 completed
2021-11-11T12:20:28.640 INFO:teuthology.kill:2021-11-11 12:20:28,639.639 INFO:teuthology.lock.ops:unlocked: smithi043.front.sepia.ceph.com
2021-11-11T12:20:32.598 INFO:teuthology.kill:2021-11-11 12:20:32,597.597 INFO:teuthology.orchestra.console:Power off for smithi092 completed
2021-11-11T12:20:32.743 INFO:teuthology.kill:2021-11-11 12:20:32,743.743 INFO:teuthology.lock.ops:unlocked: smithi092.front.sepia.ceph.com
2021-11-11T12:22:33.011 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-11T12:22:33.106 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi043 but it was locked by another job: /home/teuthworker/archive/rfriedma-2021-11-11_11:37:48-smoke-wip-rf-log-options-distro-basic-smithi/6497347
2021-11-11T12:22:33.106 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi092 but it was locked by another job: /home/teuthworker/archive/rfriedma-2021-11-11_11:37:48-smoke-wip-rf-log-options-distro-basic-smithi/6497347