2021-11-13T15:06:10.159 INFO:root:teuthology version: 1.1.0-27954452 2021-11-13T15:06:10.291 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba015.front.sepia.ceph.com' reimaging 2021-11-13T15:06:10.293 INFO:teuthology.lock.ops:Updating [ubuntu@gibba015.front.sepia.ceph.com]: reset os type and version on server 2021-11-13T15:06:10.293 INFO:teuthology.lock.ops:Updating gibba015.front.sepia.ceph.com on lock server 2021-11-13T15:06:10.315 INFO:teuthology.lock.ops:Node 'ubuntu@gibba015.front.sepia.ceph.com' reimaging is complete 2021-11-13T15:06:10.315 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba032.front.sepia.ceph.com' reimaging 2021-11-13T15:06:10.316 INFO:teuthology.lock.ops:Updating [ubuntu@gibba032.front.sepia.ceph.com]: reset os type and version on server 2021-11-13T15:06:10.316 INFO:teuthology.lock.ops:Updating gibba032.front.sepia.ceph.com on lock server 2021-11-13T15:06:10.337 INFO:teuthology.lock.ops:Node 'ubuntu@gibba032.front.sepia.ceph.com' reimaging is complete 2021-11-13T15:06:10.546 INFO:teuthology.provision.fog.gibba032:Scheduling deploy of centos 8.3 2021-11-13T15:06:10.552 INFO:teuthology.provision.fog.gibba015:Scheduling deploy of centos 8.3 2021-11-13T15:06:10.940 INFO:teuthology.orchestra.console:Power off gibba015 2021-11-13T15:06:10.960 INFO:teuthology.orchestra.console:Power off gibba032 2021-11-13T15:06:19.328 INFO:teuthology.orchestra.console:Power off for gibba015 completed 2021-11-13T15:06:19.346 INFO:teuthology.orchestra.console:Power off for gibba032 completed 2021-11-13T15:06:19.430 INFO:teuthology.orchestra.console:Power on gibba015 2021-11-13T15:06:19.448 INFO:teuthology.orchestra.console:Power on gibba032 2021-11-13T15:06:27.810 INFO:teuthology.orchestra.console:Power on for gibba015 completed 2021-11-13T15:06:27.817 INFO:teuthology.orchestra.console:Power on for gibba032 completed 2021-11-13T15:06:27.912 INFO:teuthology.provision.fog.gibba015:Waiting for deploy to finish 2021-11-13T15:06:27.918 INFO:teuthology.provision.fog.gibba032:Waiting for deploy to finish 2021-11-13T15:10:24.759 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-13T15:10:34.925 INFO:teuthology.provision.fog.gibba032:Node is ready 2021-11-13T15:10:34.941 INFO:teuthology.orchestra.run.gibba032.stdout:gibba032 2021-11-13T15:10:35.003 INFO:teuthology.orchestra.run.gibba032.stdout:172.21.2.132 gibba032.front.sepia.ceph.com gibba032 2021-11-13T15:10:35.327 INFO:teuthology.provision.fog.gibba032:Deploy complete! 2021-11-13T15:10:59.252 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-13T15:10:59.546 INFO:teuthology.provision.fog.gibba015:Node is ready 2021-11-13T15:10:59.558 INFO:teuthology.orchestra.run.gibba015.stdout:gibba015.front.sepia.ceph.com 2021-11-13T15:10:59.611 INFO:teuthology.orchestra.run.gibba015.stdout:172.21.2.115 gibba015.front.sepia.ceph.com gibba015 2021-11-13T15:10:59.943 INFO:teuthology.provision.fog.gibba015:Deploy complete! 2021-11-13T15:11:00.233 INFO:teuthology.lock.ops:Checking gibba015.front.sepia.ceph.com 2021-11-13T15:11:00.234 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-13T15:11:00.263 INFO:teuthology.lock.ops:Checking gibba032.front.sepia.ceph.com 2021-11-13T15:11:00.264 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-13T15:11:00.286 INFO:teuthology.lock.ops:Updating [gibba015.front.sepia.ceph.com]: set os type and version on server 2021-11-13T15:11:00.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba015.front.sepia.ceph.com' 2021-11-13T15:11:00.514 INFO:teuthology.orchestra.run.gibba015.stdout:x86_64 2021-11-13T15:11:00.566 INFO:teuthology.orchestra.run.gibba015.stdout:NAME="CentOS Linux" 2021-11-13T15:11:00.567 INFO:teuthology.orchestra.run.gibba015.stdout:VERSION="8" 2021-11-13T15:11:00.567 INFO:teuthology.orchestra.run.gibba015.stdout:ID="centos" 2021-11-13T15:11:00.567 INFO:teuthology.orchestra.run.gibba015.stdout:ID_LIKE="rhel fedora" 2021-11-13T15:11:00.568 INFO:teuthology.orchestra.run.gibba015.stdout:VERSION_ID="8" 2021-11-13T15:11:00.568 INFO:teuthology.orchestra.run.gibba015.stdout:PLATFORM_ID="platform:el8" 2021-11-13T15:11:00.568 INFO:teuthology.orchestra.run.gibba015.stdout:PRETTY_NAME="CentOS Linux 8" 2021-11-13T15:11:00.568 INFO:teuthology.orchestra.run.gibba015.stdout:ANSI_COLOR="0;31" 2021-11-13T15:11:00.569 INFO:teuthology.orchestra.run.gibba015.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-13T15:11:00.569 INFO:teuthology.orchestra.run.gibba015.stdout:HOME_URL="https://centos.org/" 2021-11-13T15:11:00.569 INFO:teuthology.orchestra.run.gibba015.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-11-13T15:11:00.569 INFO:teuthology.orchestra.run.gibba015.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-11-13T15:11:00.569 INFO:teuthology.orchestra.run.gibba015.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-11-13T15:11:00.571 INFO:teuthology.lock.ops:Updating gibba015.front.sepia.ceph.com on lock server 2021-11-13T15:11:00.590 INFO:teuthology.lock.ops:Updating [gibba032.front.sepia.ceph.com]: set os type and version on server 2021-11-13T15:11:00.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba032.front.sepia.ceph.com' 2021-11-13T15:11:00.819 INFO:teuthology.orchestra.run.gibba032.stdout:x86_64 2021-11-13T15:11:00.871 INFO:teuthology.orchestra.run.gibba032.stdout:NAME="CentOS Linux" 2021-11-13T15:11:00.871 INFO:teuthology.orchestra.run.gibba032.stdout:VERSION="8" 2021-11-13T15:11:00.872 INFO:teuthology.orchestra.run.gibba032.stdout:ID="centos" 2021-11-13T15:11:00.872 INFO:teuthology.orchestra.run.gibba032.stdout:ID_LIKE="rhel fedora" 2021-11-13T15:11:00.872 INFO:teuthology.orchestra.run.gibba032.stdout:VERSION_ID="8" 2021-11-13T15:11:00.872 INFO:teuthology.orchestra.run.gibba032.stdout:PLATFORM_ID="platform:el8" 2021-11-13T15:11:00.872 INFO:teuthology.orchestra.run.gibba032.stdout:PRETTY_NAME="CentOS Linux 8" 2021-11-13T15:11:00.873 INFO:teuthology.orchestra.run.gibba032.stdout:ANSI_COLOR="0;31" 2021-11-13T15:11:00.873 INFO:teuthology.orchestra.run.gibba032.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-13T15:11:00.873 INFO:teuthology.orchestra.run.gibba032.stdout:HOME_URL="https://centos.org/" 2021-11-13T15:11:00.873 INFO:teuthology.orchestra.run.gibba032.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-11-13T15:11:00.874 INFO:teuthology.orchestra.run.gibba032.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-11-13T15:11:00.874 INFO:teuthology.orchestra.run.gibba032.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-11-13T15:11:00.874 INFO:teuthology.lock.ops:Updating gibba032.front.sepia.ceph.com on lock server 2021-11-13T15:11:00.973 INFO:teuthology.dispatcher.supervisor:Running job 6489605 2021-11-13T15:11:00.974 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-11-07_07:01:02-rados-master-distro-default-gibba/6489605 --name teuthology-2021-11-07_07:01:02-rados-master-distro-default-gibba --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/teuthology-2021-11-07_07:01:02-rados-master-distro-default-gibba/6489605/orig.config.yaml 2021-11-13T15:11:00.977 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-11-07_07:01:02-rados-master-distro-default-gibba/6489605 2021-11-13T15:11:00.978 INFO:teuthology.dispatcher.supervisor:Job PID: 18285 2021-11-13T15:11:00.979 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-14T03:11:56.427 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-14T03:11:56.570 INFO:teuthology.kill:Killing Pids: {18285} 2021-11-14T03:11:57.781 INFO:teuthology.task.internal:roles: ubuntu@gibba015.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-11-14T03:11:57.782 INFO:teuthology.task.internal:roles: ubuntu@gibba032.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-11-14T03:11:57.782 INFO:teuthology.misc:Compressing logs... 2021-11-14T03:11:57.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba015.front.sepia.ceph.com' 2021-11-14T03:11:57.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba032.front.sepia.ceph.com' 2021-11-14T03:11:58.230 INFO:teuthology.misc:Compressing logs... 2021-11-14T03:11:58.503 INFO:teuthology.misc:Compressing logs... 2021-11-14T03:12:04.586 INFO:teuthology.orchestra.run.gibba032.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-mon.gibba032.log: file size changed while zipping 2021-11-14T03:12:17.252 INFO:teuthology.orchestra.run.gibba015.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-mon.gibba015.log: file size changed while zipping 2021-11-14T03:12:19.991 INFO:teuthology.orchestra.run.gibba032.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-11-14T03:12:32.271 INFO:teuthology.orchestra.run.gibba032.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.gibba032.ytkpqe.log: file size changed while zipping 2021-11-14T03:12:52.425 INFO:teuthology.orchestra.run.gibba032.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.gibba032.hrefyj.log: file size changed while zipping 2021-11-14T03:12:53.663 INFO:teuthology.orchestra.run.gibba015.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-mds.cephfs.gibba015.cvgffi.log: file size changed while zipping 2021-11-14T03:13:04.801 INFO:teuthology.orchestra.run.gibba032.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-11-14T03:13:07.530 INFO:teuthology.orchestra.run.gibba015.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-11-14T03:13:16.074 INFO:teuthology.orchestra.run.gibba015.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-mgr.gibba015.rthbgf.log: file size changed while zipping 2021-11-14T03:13:18.606 INFO:teuthology.orchestra.run.gibba032.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-11-14T03:13:29.717 INFO:teuthology.orchestra.run.gibba015.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-11-14T03:13:48.177 INFO:teuthology.orchestra.run.gibba015.stderr:gzip: /var/log/ceph/59bd263c-4495-11ec-8c2c-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-11-14T03:14:27.517 INFO:teuthology.kill:No teuthology processes running 2021-11-14T03:14:28.651 INFO:teuthology.kill:Nuking machines: ['gibba015', 'gibba032'] 2021-11-14T03:14:29.618 INFO:teuthology.kill:2021-11-14 03:14:29,618.618 INFO:teuthology.nuke:targets: 2021-11-14T03:14:29.619 INFO:teuthology.kill: gibba015.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPCKFjSJurVW+4OIfVkvvOfGiwimhTgKG40dsETZtVMBjxIL9s/Yrv2+GYas/hUGQdQEdohMzQSZa2kQU+Nv2nVWGWGsTlEyaAkaiLmSJhgdgHHggK2Gpc23W6+tOK4LU3s03lNXVumQX1QO2DUjfBXKrOiED1RO8NZCQbrbBPhK8mCWImn8cPd/sWfCtiLs3yHLDOdx6CGQt8B3S1avSSYIdXoVMk8LhZeySnAOZFkv1JZf/JEldRPXinbjj8sUOFiiN+eY0DDVkQ6zIwR+RE2+0wYOD7/5jxBp1SMuZ0wQbPXeMBOOm7ge4LUNfN/LnWv9qIm3HU13DEdPBKq02X+si0Ya6Faz2vFhGzqiZkDf66NMXwwEGeuUh1GEcgp5oeKe3YLSuwj9BIRU0hPftBKtrVS+kGyn+KyLWKRIPQ0UeovR8KPrRhdDBTMUcWRiNeNhYSAOPwq5Vw8+iY7lRnYbRKJa1TMrUNb7fg/Ju1eVmN+2WvVhAx1pCdI90o6r0= 2021-11-14T03:14:29.619 INFO:teuthology.kill: gibba032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkAPsW1GrhgKutnpBjN2cIDtlfixrRE6+H41IYk2FQOFmb++JQGN1U7po8clkp+JuBNF5G72r3pzajCCBybbPwa+ZcOqMV931hlYZwIflYk+9SBW9RUuWtujsre2LmEIYNmArzc/vl6tJeaAUnX1dSfAUKOf9Vb3MQQu/7d2JwVytlBKCmPHuLey3BflqZ9O/01egkCDphUTmlv+bi/tttkOuSk71pPQGsWh2xG+sCrX3mmwMFZaonkQBdgl9y43zEubbrrz64y8L/6onv9O2gymhmzrIhZzjH58G6/d0V9Ex2dMjTHwy4SY0ntg8S8M1L9WVz5VcRHf+FcNUlwbuGdiuYNzKEIooQZ/+6/DJDPYhRKaMs4PqrwdyIKGHAUH7+JFiUHKeuqs9OIW6WfpbZfqYee+hw2TUjII7/DxKgdflvB7uIzDsIqw/ySJ0cU6Kr+RSwOxDE0U1zlyTK/Kd3R6GXu4EccjVghe+3jYKh2x/oXsC81qkiKDiyUlbfJEc= 2021-11-14T03:14:29.645 INFO:teuthology.kill:2021-11-14 03:14:29,645.645 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-14T03:14:29.649 INFO:teuthology.kill:2021-11-14 03:14:29,649.649 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-14T03:14:29.728 INFO:teuthology.kill:2021-11-14 03:14:29,728.728 INFO:teuthology.orchestra.console:Power off gibba032 2021-11-14T03:14:29.740 INFO:teuthology.kill:2021-11-14 03:14:29,740.740 INFO:teuthology.orchestra.console:Power off gibba015 2021-11-14T03:14:38.110 INFO:teuthology.kill:2021-11-14 03:14:38,109.109 INFO:teuthology.orchestra.console:Power off for gibba015 completed 2021-11-14T03:14:38.257 INFO:teuthology.kill:2021-11-14 03:14:38,256.256 INFO:teuthology.lock.ops:unlocked: gibba015.front.sepia.ceph.com 2021-11-14T03:14:42.211 INFO:teuthology.kill:2021-11-14 03:14:42,211.211 INFO:teuthology.orchestra.console:Power off for gibba032 completed 2021-11-14T03:14:42.357 INFO:teuthology.kill:2021-11-14 03:14:42,357.357 INFO:teuthology.lock.ops:unlocked: gibba032.front.sepia.ceph.com 2021-11-14T03:16:42.660 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-14T03:16:42.774 WARNING:teuthology.dispatcher.supervisor:Was going to unlock gibba032 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-11-10_03:30:03-rados-octopus-distro-default-gibba/6495181