2021-11-08T13:00:47.685 INFO:root:teuthology version: 1.1.0-c56135d1
2021-11-08T13:00:47.815 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi081.front.sepia.ceph.com' reimaging
2021-11-08T13:00:47.816 INFO:teuthology.lock.ops:Updating [ubuntu@smithi081.front.sepia.ceph.com]: reset os type and version on server
2021-11-08T13:00:47.817 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server
2021-11-08T13:00:47.836 INFO:teuthology.lock.ops:Node 'ubuntu@smithi081.front.sepia.ceph.com' reimaging is complete
2021-11-08T13:00:47.837 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi117.front.sepia.ceph.com' reimaging
2021-11-08T13:00:47.837 INFO:teuthology.lock.ops:Updating [ubuntu@smithi117.front.sepia.ceph.com]: reset os type and version on server
2021-11-08T13:00:47.838 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server
2021-11-08T13:00:47.857 INFO:teuthology.lock.ops:Node 'ubuntu@smithi117.front.sepia.ceph.com' reimaging is complete
2021-11-08T13:00:48.054 INFO:teuthology.provision.fog.smithi081:Scheduling deploy of centos 8.3
2021-11-08T13:00:48.079 INFO:teuthology.provision.fog.smithi117:Scheduling deploy of centos 8.3
2021-11-08T13:00:48.403 INFO:teuthology.orchestra.console:Power off smithi081
2021-11-08T13:00:48.422 INFO:teuthology.orchestra.console:Power off smithi117
2021-11-08T13:00:56.791 INFO:teuthology.orchestra.console:Power off for smithi117 completed
2021-11-08T13:00:56.892 INFO:teuthology.orchestra.console:Power on smithi117
2021-11-08T13:01:00.892 INFO:teuthology.orchestra.console:Power off for smithi081 completed
2021-11-08T13:01:00.995 INFO:teuthology.orchestra.console:Power on smithi081
2021-11-08T13:01:05.249 INFO:teuthology.orchestra.console:Power on for smithi081 completed
2021-11-08T13:01:05.255 INFO:teuthology.orchestra.console:Power on for smithi117 completed
2021-11-08T13:01:05.351 INFO:teuthology.provision.fog.smithi081:Waiting for deploy to finish
2021-11-08T13:01:05.357 INFO:teuthology.provision.fog.smithi117:Waiting for deploy to finish
2021-11-08T13:05:54.991 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-08T13:05:55.020 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-08T13:05:55.282 INFO:teuthology.provision.fog.smithi117:Node is ready
2021-11-08T13:05:55.297 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117.front.sepia.ceph.com
2021-11-08T13:05:55.340 INFO:teuthology.provision.fog.smithi081:Node is ready
2021-11-08T13:05:55.353 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117 smithi117.front.sepia.ceph.com smithi117
2021-11-08T13:05:55.355 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081.front.sepia.ceph.com
2021-11-08T13:05:55.410 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81 smithi081.front.sepia.ceph.com smithi081
2021-11-08T13:05:55.686 INFO:teuthology.provision.fog.smithi117:Deploy complete!
2021-11-08T13:05:55.722 INFO:teuthology.provision.fog.smithi081:Deploy complete!
2021-11-08T13:05:55.883 INFO:teuthology.lock.ops:Checking smithi081.front.sepia.ceph.com
2021-11-08T13:05:55.884 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-08T13:05:55.905 INFO:teuthology.lock.ops:Checking smithi117.front.sepia.ceph.com
2021-11-08T13:05:55.905 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-08T13:05:55.924 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: set os type and version on server
2021-11-08T13:05:55.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2021-11-08T13:05:56.178 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64
2021-11-08T13:05:56.231 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Linux"
2021-11-08T13:05:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="8"
2021-11-08T13:05:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos"
2021-11-08T13:05:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora"
2021-11-08T13:05:56.233 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="8"
2021-11-08T13:05:56.233 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el8"
2021-11-08T13:05:56.233 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-08T13:05:56.233 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31"
2021-11-08T13:05:56.234 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-08T13:05:56.234 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/"
2021-11-08T13:05:56.234 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-08T13:05:56.234 INFO:teuthology.orchestra.run.smithi081.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-08T13:05:56.235 INFO:teuthology.orchestra.run.smithi081.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-08T13:05:56.236 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server
2021-11-08T13:05:56.255 INFO:teuthology.lock.ops:Updating [smithi117.front.sepia.ceph.com]: set os type and version on server
2021-11-08T13:05:56.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com'
2021-11-08T13:05:56.511 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64
2021-11-08T13:05:56.565 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Linux"
2021-11-08T13:05:56.565 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="8"
2021-11-08T13:05:56.566 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos"
2021-11-08T13:05:56.566 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora"
2021-11-08T13:05:56.566 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="8"
2021-11-08T13:05:56.566 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el8"
2021-11-08T13:05:56.567 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-08T13:05:56.567 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31"
2021-11-08T13:05:56.567 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-08T13:05:56.567 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/"
2021-11-08T13:05:56.567 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-08T13:05:56.568 INFO:teuthology.orchestra.run.smithi117.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-08T13:05:56.568 INFO:teuthology.orchestra.run.smithi117.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-08T13:05:56.569 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server
2021-11-08T13:05:56.672 INFO:teuthology.dispatcher.supervisor:Running job 6484614
2021-11-08T13:05:56.672 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_akupczyk@teuthology --archive /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484614 --name akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484614/orig.config.yaml
2021-11-08T13:05:56.676 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484614
2021-11-08T13:05:56.677 INFO:teuthology.dispatcher.supervisor:Job PID: 2001
2021-11-08T13:05:56.677 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-09T01:06:53.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-09T01:06:54.630 INFO:teuthology.kill:Killing Pids: {2001}
2021-11-09T01:06:55.940 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-09T01:06:55.943 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-09T01:06:55.950 INFO:teuthology.misc:Compressing logs...
2021-11-09T01:06:55.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2021-11-09T01:06:56.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com'
2021-11-09T01:06:57.080 INFO:teuthology.misc:Compressing logs...
2021-11-09T01:06:57.420 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-09T01:06:57.551 INFO:teuthology.orchestra.run.smithi117.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-09T01:06:57.887 INFO:teuthology.misc:Compressing logs...
2021-11-09T01:07:06.473 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-mgr.smithi081.mouuyq.log: file size changed while zipping
2021-11-09T01:07:20.423 INFO:teuthology.orchestra.run.smithi117.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-11-09T01:07:26.681 INFO:teuthology.orchestra.run.smithi117.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-mon.smithi117.log: file size changed while zipping
2021-11-09T01:07:51.763 INFO:teuthology.orchestra.run.smithi117.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-11-09T01:08:04.177 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.smithi081.dsxyxu.log: file size changed while zipping
2021-11-09T01:08:11.718 INFO:teuthology.orchestra.run.smithi117.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-11-09T01:08:37.287 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-mon.smithi081.log: file size changed while zipping
2021-11-09T01:08:54.599 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-11-09T01:08:55.007 INFO:teuthology.orchestra.run.smithi117.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.smithi117.itizgb.log: file size changed while zipping
2021-11-09T01:09:16.411 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /var/log/ceph/450f0c44-4096-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-11-09T01:10:10.924 INFO:teuthology.kill:No teuthology processes running
2021-11-09T01:10:12.190 INFO:teuthology.kill:Nuking machines: ['smithi081', 'smithi117']
2021-11-09T01:10:13.158 INFO:teuthology.kill:2021-11-09 01:10:13,158.158 INFO:teuthology.nuke:targets:
2021-11-09T01:10:13.159 INFO:teuthology.kill:  smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhGLNBdPC4JeNheDqj338jnPuSCjFC3nJvgQehgIHbImSVU2joGDR9a+nOURV75Ly4H35HgeCREV6JePvXIghk3obftwYZXkgRyMZCR+wYebl/7SzGiIfv9kQd7joT1eikwTlj5sXJGnt9bgFRBgOc7pNB4k9I4whAJhxQgHXpZu0T84XGLRSsthOQuaSlzWjkUQfWdbIulEnq2FRIBFLMwORQvzVbFm/SUwpAD7Rk0hR9jAxS6UwSWRdj9llE6tLnkji2RzGsAiOd7yWlCrxCsrvt18vS5c/iKO2lrZQOBP3RKoTFXfzZiLBxQJMcOdRngw+X+x68Sp2//lUarOplVbzYnOormT2Jnyr5p7USz98PdpIaWVDU3dRGl7GJGrWG2FExrsck0r/JxneCOJK5JrshlMxnCgdEABHhSZEb++C6gWKlMvlg3WhqmQrMutq3417AxQ7g+fFmXG535QwTLHPdcBOvUCtrFlhlz/GoZt77pLmgyMk1nJ5Z6aOaqvk=
2021-11-09T01:10:13.160 INFO:teuthology.kill:  smithi117.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDesjNZULqT15XMn6Kmzm+waxRTb54oFM8ExyYjAO9AifCiDMJuZQhXjnZm+LOrEOxugHSc1pq0513Cz2bMe7UXFmK/YZwmDywSPQxgq8NU5NcOBytVYeW+EgGhHXm/PINefkyNnuylut2k4ujyC2JeI3Hh2YT3fpajjqjZxb0HJaYE1kaG4MixC918+YR/loz+NbeX7bEh74qcToH4AldM4Ts624wGkd36PjMTUweWRtNNkR5Tey8IWt5FS/yeFVkZnXy5hFMRP6pZdEK9aFAs85Eb5/HfEYBrU8JeuCS4CdmZ9w/Q0ZtTNgeQvILpBlwEmucmCHsbg1Ua9+UQkTQ6iL5YOaidC+ob+pIgqslUNehD+e1bxiDo+4g7MM5IbOpEIasqhMOyjOxkiMS3hJcb1J/PQ59ygAKRkInshH/urgYls4kuw8GC0eC4SLu7zQHhrOOIZtM3TuXO0Q2Cq2I/b5j6a+l6ayqfUTenn++sY1REu0aLUgsHHnIQO+yBcz8=
2021-11-09T01:10:13.189 INFO:teuthology.kill:2021-11-09 01:10:13,189.189 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-09T01:10:13.191 INFO:teuthology.kill:2021-11-09 01:10:13,191.191 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-09T01:10:13.275 INFO:teuthology.kill:2021-11-09 01:10:13,274.274 INFO:teuthology.orchestra.console:Power off smithi081
2021-11-09T01:10:13.290 INFO:teuthology.kill:2021-11-09 01:10:13,290.290 INFO:teuthology.orchestra.console:Power off smithi117
2021-11-09T01:10:21.657 INFO:teuthology.kill:2021-11-09 01:10:21,657.657 INFO:teuthology.orchestra.console:Power off for smithi117 completed
2021-11-09T01:10:21.805 INFO:teuthology.kill:2021-11-09 01:10:21,804.804 INFO:teuthology.lock.ops:unlocked: smithi117.front.sepia.ceph.com
2021-11-09T01:10:25.764 INFO:teuthology.kill:2021-11-09 01:10:25,764.764 INFO:teuthology.orchestra.console:Power off for smithi081 completed
2021-11-09T01:10:25.910 INFO:teuthology.kill:2021-11-09 01:10:25,909.909 INFO:teuthology.lock.ops:unlocked: smithi081.front.sepia.ceph.com
2021-11-09T01:12:26.226 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-09T01:12:26.353 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi117 but it was locked by another job: /home/teuthworker/archive/ozeneva-2021-11-08_23:48:28-rados-wip-omri-tracer-opentelemetry-distro-basic-smithi/6492819