2021-11-10T17:47:57.165 INFO:root:teuthology version: 1.1.0-c56135d1
2021-11-10T17:47:57.302 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi121.front.sepia.ceph.com' reimaging
2021-11-10T17:47:57.303 INFO:teuthology.lock.ops:Updating [ubuntu@smithi121.front.sepia.ceph.com]: reset os type and version on server
2021-11-10T17:47:57.303 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server
2021-11-10T17:47:57.322 INFO:teuthology.lock.ops:Node 'ubuntu@smithi121.front.sepia.ceph.com' reimaging is complete
2021-11-10T17:47:57.322 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging
2021-11-10T17:47:57.323 INFO:teuthology.lock.ops:Updating [ubuntu@smithi187.front.sepia.ceph.com]: reset os type and version on server
2021-11-10T17:47:57.323 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server
2021-11-10T17:47:57.342 INFO:teuthology.lock.ops:Node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging is complete
2021-11-10T17:47:57.530 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of centos 8.3
2021-11-10T17:47:57.567 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of centos 8.3
2021-11-10T17:47:57.932 INFO:teuthology.orchestra.console:Power off smithi121
2021-11-10T17:47:57.951 INFO:teuthology.orchestra.console:Power off smithi187
2021-11-10T17:48:06.309 INFO:teuthology.orchestra.console:Power off for smithi121 completed
2021-11-10T17:48:06.410 INFO:teuthology.orchestra.console:Power on smithi121
2021-11-10T17:48:10.427 INFO:teuthology.orchestra.console:Power off for smithi187 completed
2021-11-10T17:48:10.529 INFO:teuthology.orchestra.console:Power on smithi187
2021-11-10T17:48:10.687 INFO:teuthology.orchestra.console:Power on for smithi121 completed
2021-11-10T17:48:10.788 INFO:teuthology.provision.fog.smithi121:Waiting for deploy to finish
2021-11-10T17:48:18.990 INFO:teuthology.orchestra.console:Power on for smithi187 completed
2021-11-10T17:48:19.091 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish
2021-11-10T17:53:00.463 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-10T17:53:00.776 INFO:teuthology.provision.fog.smithi121:Node is ready
2021-11-10T17:53:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121.front.sepia.ceph.com
2021-11-10T17:53:00.847 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121 smithi121.front.sepia.ceph.com smithi121
2021-11-10T17:53:01.164 INFO:teuthology.provision.fog.smithi121:Deploy complete!
2021-11-10T17:53:08.698 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-10T17:53:08.995 INFO:teuthology.provision.fog.smithi187:Node is ready
2021-11-10T17:53:09.010 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com
2021-11-10T17:53:09.064 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187
2021-11-10T17:53:09.396 INFO:teuthology.provision.fog.smithi187:Deploy complete!
2021-11-10T17:53:09.543 INFO:teuthology.lock.ops:Checking smithi121.front.sepia.ceph.com
2021-11-10T17:53:09.544 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-10T17:53:09.563 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com
2021-11-10T17:53:09.563 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-10T17:53:09.580 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: set os type and version on server
2021-11-10T17:53:09.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com'
2021-11-10T17:53:09.852 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64
2021-11-10T17:53:09.906 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Linux"
2021-11-10T17:53:09.906 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8"
2021-11-10T17:53:09.907 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos"
2021-11-10T17:53:09.907 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora"
2021-11-10T17:53:09.907 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8"
2021-11-10T17:53:09.907 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8"
2021-11-10T17:53:09.907 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-10T17:53:09.908 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31"
2021-11-10T17:53:09.908 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-10T17:53:09.908 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/"
2021-11-10T17:53:09.908 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-10T17:53:09.909 INFO:teuthology.orchestra.run.smithi121.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-10T17:53:09.909 INFO:teuthology.orchestra.run.smithi121.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-10T17:53:09.910 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server
2021-11-10T17:53:09.928 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server
2021-11-10T17:53:09.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2021-11-10T17:53:10.128 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64
2021-11-10T17:53:10.181 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Linux"
2021-11-10T17:53:10.182 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8"
2021-11-10T17:53:10.182 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos"
2021-11-10T17:53:10.182 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora"
2021-11-10T17:53:10.182 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8"
2021-11-10T17:53:10.183 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8"
2021-11-10T17:53:10.183 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-10T17:53:10.183 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31"
2021-11-10T17:53:10.183 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-10T17:53:10.183 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/"
2021-11-10T17:53:10.184 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-10T17:53:10.184 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-10T17:53:10.184 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-10T17:53:10.185 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server
2021-11-10T17:53:10.282 INFO:teuthology.dispatcher.supervisor:Running job 6484833
2021-11-10T17:53:10.282 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_akupczyk@teuthology --archive /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484833 --name akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484833/orig.config.yaml
2021-11-10T17:53:10.285 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484833
2021-11-10T17:53:10.286 INFO:teuthology.dispatcher.supervisor:Job PID: 14573
2021-11-10T17:53:10.286 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-11T05:54:03.273 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-11T05:54:03.483 INFO:teuthology.kill:Killing Pids: {14573}
2021-11-11T05:54:04.694 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-11T05:54:04.695 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-11T05:54:04.695 INFO:teuthology.misc:Compressing logs...
2021-11-11T05:54:04.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com'
2021-11-11T05:54:04.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2021-11-11T05:54:05.265 INFO:teuthology.misc:Compressing logs...
2021-11-11T05:54:05.623 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-11T05:54:05.749 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-11T05:54:05.998 INFO:teuthology.misc:Compressing logs...
2021-11-11T05:54:07.710 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-mgr.smithi121.iqtmso.log: file size changed while zipping
2021-11-11T05:54:09.717 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-11-11T05:54:13.558 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-mon.smithi121.log: file size changed while zipping
2021-11-11T05:54:14.049 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-11-11T05:54:15.513 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-mon.smithi187.log: file size changed while zipping
2021-11-11T05:54:15.779 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.smithi187.leglgu.log: file size changed while zipping
2021-11-11T05:54:16.186 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-11-11T05:54:16.206 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.smithi121.wrjwnx.log: file size changed while zipping
2021-11-11T05:54:16.839 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.smithi121.rnvlbk.log: file size changed while zipping
2021-11-11T05:54:19.345 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-11-11T05:54:20.143 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-11-11T05:54:54.277 INFO:teuthology.orchestra.run.smithi187.stderr:tar: ./d7b6503c-4250-11ec-8c2c-001a4aab830c/ceph-osd.3.log.gz: file changed as we read it
2021-11-11T05:55:15.185 INFO:teuthology.orchestra.run.smithi187.stderr:tar: ./d7b6503c-4250-11ec-8c2c-001a4aab830c: file changed as we read it
2021-11-11T05:55:15.186 INFO:teuthology.orchestra.run.smithi187.stderr:tar: .: file changed as we read it
2021-11-11T05:55:15.273 INFO:teuthology.kill:No teuthology processes running
2021-11-11T05:55:16.495 INFO:teuthology.kill:Nuking machines: ['smithi121', 'smithi187']
2021-11-11T05:55:17.439 INFO:teuthology.kill:2021-11-11 05:55:17,439.439 INFO:teuthology.nuke:targets:
2021-11-11T05:55:17.440 INFO:teuthology.kill:  smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCV0DR/xTWTXOFfGe6LNsbTpcn6vprJg8dwl1+zDSiL0aFrTBBds9AiBhd3x1ZK+nDhEfjE81uO6okcNw6AIfySGvvu5HyEMTqi/0wROJicv8Pbutufz8Lx4AhZzc07rYS+0Qns5MpqAj6wB3WcFy7/LTj0TCyQXAX0rVvhnqdioPoKuCBMoGNZwFXMaFF0B5jst5BNAtZ9a4NSVARFJZttVnVUOv9rL4MpQqu18rOrDQRsZbs9mNiY4bjhc+tH3EZlDLcWGhVccdHdlUWl3LRtrJ6MtXLKEvfZB5TJOkBEi22Pcw1Ek94pl3qxSoqxx/uES46iPiE/TinuUtNKca8+OIoGJIuJ3uDtJOYa3atvhTIOM+Dn7cMLuaQerxJgMXbxzTFmkNmKEjqvX5YO2U5NQXibd2dYQYos93Vj1VyVxI0r+ChscyZpv+75Y9sVEqnx/ladmMmsRyfTGQsdYPNwbNL03lZAeYqW7TbgbQLLaLA0HAEBkjlTe5V78shd36E=
2021-11-11T05:55:17.441 INFO:teuthology.kill:  smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgtBL3n2WjqyWq8f5npdemacvJhnDe31Te7EwENgjlAUT5BKYHlUiHX8fXI4h1eY1W/wntkBOOzDylP6JEDW6zTqrtbvm1gtJxmhbLjRrf3AmE2BCrb27+TvMFea7sHL594ZJ6ptZKzfUu5mYdYFBKRrXFt7HqBR58OghaT1bF7rmoMtnL/Gf6NKsuwINlD48byQ/TlhF3bv+xNGpCGdmseaK66YdXQaqmODjXSdNRqdI1m+LuFEFQ6/CfJX057Nlxo7xDPULsY741m9RWQeucwS2YEF0DsdryZzibSKeH7t8Rht6BCRiK2wduUSl61LjQTSJobLDgMQ3Bb4nRx5o8BeZ8FbsTjPVP8KYVyn5yV4zAq6Sb4FncuFueXMMs/ci2I+xF+Q1Jm8jzMDPsv6agysKX9SpPdzte/fZ7N+2o/TDG75MrHqyKsP63cjyLVjSjwEvgsnLZoGGHfHY9N9wttCeYfsTqGCkXVNDZ1vV5C87Dn+lAuKMDDUYWsv8v3Dk=
2021-11-11T05:55:17.466 INFO:teuthology.kill:2021-11-11 05:55:17,466.466 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T05:55:17.469 INFO:teuthology.kill:2021-11-11 05:55:17,469.469 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T05:55:17.549 INFO:teuthology.kill:2021-11-11 05:55:17,549.549 INFO:teuthology.orchestra.console:Power off smithi187
2021-11-11T05:55:17.562 INFO:teuthology.kill:2021-11-11 05:55:17,561.561 INFO:teuthology.orchestra.console:Power off smithi121
2021-11-11T05:55:25.917 INFO:teuthology.kill:2021-11-11 05:55:25,917.917 INFO:teuthology.orchestra.console:Power off for smithi187 completed
2021-11-11T05:55:26.067 INFO:teuthology.kill:2021-11-11 05:55:26,066.066 INFO:teuthology.lock.ops:unlocked: smithi187.front.sepia.ceph.com
2021-11-11T05:55:30.039 INFO:teuthology.kill:2021-11-11 05:55:30,039.039 INFO:teuthology.orchestra.console:Power off for smithi121 completed
2021-11-11T05:55:30.188 INFO:teuthology.kill:2021-11-11 05:55:30,187.187 INFO:teuthology.lock.ops:unlocked: smithi121.front.sepia.ceph.com
2021-11-11T05:57:30.466 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-11T05:57:30.583 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi121 but it was locked by another job: /home/teuthworker/archive/soumyakoduri-2021-11-11_04:48:08-rgw-cloudtiering-distro-basic-smithi/6496838
2021-11-11T05:57:30.584 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi187 but it was locked by another job: /home/teuthworker/archive/soumyakoduri-2021-11-11_04:48:08-rgw-cloudtiering-distro-basic-smithi/6496838