2021-12-18T05:52:06.843 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-18T05:52:06.972 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging 2021-12-18T05:52:06.974 INFO:teuthology.lock.ops:Updating [ubuntu@smithi085.front.sepia.ceph.com]: reset os type and version on server 2021-12-18T05:52:06.974 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-12-18T05:52:06.994 INFO:teuthology.lock.ops:Node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging is complete 2021-12-18T05:52:06.994 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging 2021-12-18T05:52:06.995 INFO:teuthology.lock.ops:Updating [ubuntu@smithi100.front.sepia.ceph.com]: reset os type and version on server 2021-12-18T05:52:06.995 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2021-12-18T05:52:07.015 INFO:teuthology.lock.ops:Node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging is complete 2021-12-18T05:52:07.212 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of centos 8.stream 2021-12-18T05:52:07.221 INFO:teuthology.provision.fog.smithi100:Scheduling deploy of centos 8.stream 2021-12-18T05:52:07.557 INFO:teuthology.orchestra.console:Power off smithi100 2021-12-18T05:52:07.576 INFO:teuthology.orchestra.console:Power off smithi085 2021-12-18T05:52:15.941 INFO:teuthology.orchestra.console:Power off for smithi100 completed 2021-12-18T05:52:16.042 INFO:teuthology.orchestra.console:Power on smithi100 2021-12-18T05:52:20.056 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-12-18T05:52:20.157 INFO:teuthology.orchestra.console:Power on smithi085 2021-12-18T05:52:20.295 INFO:teuthology.orchestra.console:Power on for smithi100 completed 2021-12-18T05:52:20.396 INFO:teuthology.provision.fog.smithi100:Waiting for deploy to finish 2021-12-18T05:52:24.409 INFO:teuthology.orchestra.console:Power on for smithi085 completed 2021-12-18T05:52:24.510 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish 2021-12-18T05:57:10.128 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-18T05:57:10.406 INFO:teuthology.provision.fog.smithi100:Node is ready 2021-12-18T05:57:10.423 INFO:teuthology.orchestra.run.smithi100.stdout:smithi100.front.sepia.ceph.com 2021-12-18T05:57:10.480 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.15.100 smithi100.front.sepia.ceph.com smithi100 2021-12-18T05:57:10.801 INFO:teuthology.provision.fog.smithi100:Deploy complete! 2021-12-18T05:57:14.037 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-18T05:57:14.322 INFO:teuthology.provision.fog.smithi085:Node is ready 2021-12-18T05:57:14.342 INFO:teuthology.orchestra.run.smithi085.stdout:smithi085.front.sepia.ceph.com 2021-12-18T05:57:14.402 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.85 smithi085.front.sepia.ceph.com smithi085 2021-12-18T05:57:14.751 INFO:teuthology.provision.fog.smithi085:Deploy complete! 2021-12-18T05:57:14.911 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com 2021-12-18T05:57:14.912 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-18T05:57:14.932 INFO:teuthology.lock.ops:Checking smithi100.front.sepia.ceph.com 2021-12-18T05:57:14.932 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-18T05:57:14.952 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server 2021-12-18T05:57:14.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2021-12-18T05:57:15.141 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2021-12-18T05:57:15.196 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="CentOS Stream" 2021-12-18T05:57:15.196 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="8" 2021-12-18T05:57:15.196 INFO:teuthology.orchestra.run.smithi085.stdout:ID="centos" 2021-12-18T05:57:15.197 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE="rhel fedora" 2021-12-18T05:57:15.197 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="8" 2021-12-18T05:57:15.197 INFO:teuthology.orchestra.run.smithi085.stdout:PLATFORM_ID="platform:el8" 2021-12-18T05:57:15.197 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-18T05:57:15.197 INFO:teuthology.orchestra.run.smithi085.stdout:ANSI_COLOR="0;31" 2021-12-18T05:57:15.198 INFO:teuthology.orchestra.run.smithi085.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-18T05:57:15.198 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://centos.org/" 2021-12-18T05:57:15.198 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-18T05:57:15.198 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-18T05:57:15.199 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-18T05:57:15.200 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-12-18T05:57:15.217 INFO:teuthology.lock.ops:Updating [smithi100.front.sepia.ceph.com]: set os type and version on server 2021-12-18T05:57:15.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2021-12-18T05:57:15.418 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2021-12-18T05:57:15.475 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2021-12-18T05:57:15.475 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="8" 2021-12-18T05:57:15.475 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2021-12-18T05:57:15.476 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2021-12-18T05:57:15.476 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="8" 2021-12-18T05:57:15.476 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el8" 2021-12-18T05:57:15.476 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-18T05:57:15.477 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2021-12-18T05:57:15.477 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-18T05:57:15.477 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2021-12-18T05:57:15.477 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-18T05:57:15.477 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-18T05:57:15.478 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-18T05:57:15.478 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2021-12-18T05:57:15.580 INFO:teuthology.dispatcher.supervisor:Running job 6568536 2021-12-18T05:57:15.580 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-12-17_18:08:35-rados-wip-sage2-testing-2021-12-17-1000-distro-basic-smithi/6568536 --name sage-2021-12-17_18:08:35-rados-wip-sage2-testing-2021-12-17-1000-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/sage-2021-12-17_18:08:35-rados-wip-sage2-testing-2021-12-17-1000-distro-basic-smithi/6568536/orig.config.yaml 2021-12-18T05:57:15.584 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-12-17_18:08:35-rados-wip-sage2-testing-2021-12-17-1000-distro-basic-smithi/6568536 2021-12-18T05:57:15.585 INFO:teuthology.dispatcher.supervisor:Job PID: 28023 2021-12-18T05:57:15.585 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-12-18T17:58:06.672 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-12-18T17:58:06.929 INFO:teuthology.kill:Killing Pids: {28023} 2021-12-18T17:58:08.105 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-12-18T17:58:08.106 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-12-18T17:58:08.107 INFO:teuthology.misc:Compressing logs... 2021-12-18T17:58:08.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com' 2021-12-18T17:58:08.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2021-12-18T17:58:08.600 INFO:teuthology.misc:Compressing logs... 2021-12-18T17:58:08.804 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-18T17:58:09.792 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-18T17:58:10.193 INFO:teuthology.misc:Compressing logs... 2021-12-18T17:58:18.594 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-mon.smithi100.log: file size changed while zipping 2021-12-18T17:58:25.947 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-12-18T17:58:34.717 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-12-18T17:58:45.863 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-mon.smithi085.log: file size changed while zipping 2021-12-18T17:58:53.652 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-12-18T17:58:59.666 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-12-18T17:59:16.325 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-12-18T17:59:23.476 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-mgr.smithi085.miirzt.log: file size changed while zipping 2021-12-18T17:59:39.318 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-mds.cephfs.smithi100.qfuaan.log: file size changed while zipping 2021-12-18T17:59:53.986 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-12-18T18:00:19.716 INFO:teuthology.orchestra.run.smithi085.stderr:gzip: /var/log/ceph/7071da5a-5fc8-11ec-8c31-001a4aab830c/ceph-mds.cephfs.smithi085.saivif.log: file size changed while zipping 2021-12-18T18:01:08.384 INFO:teuthology.kill:No teuthology processes running 2021-12-18T18:01:09.455 INFO:teuthology.kill:Nuking machines: ['smithi085', 'smithi100'] 2021-12-18T18:01:10.364 INFO:teuthology.kill:2021-12-18 18:01:10,364.364 INFO:teuthology.nuke:targets: 2021-12-18T18:01:10.365 INFO:teuthology.kill: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1o2PrsswBbn6uwNXtCh1gZNM4MHYaFWAlbcVICvN/Uht0ltsDdR7RmaKKVSSQdoQWZNSL2pIlwURJU9eYmZ7sFba/xJyOx0jQn3mI2HihrZUidiiUZoRyobae8J4Z4Qgu4uzVe3gFbVQk21AS+N3vh4ZVwnE7zbHkxM+34HGdFBGrwZGoZ4+qhK1WGMmVd+pIfwlejZFCYourV2+kLNUN/5rDFWUBa8mllPQ2abs3lH7a4oQN6oF8Nxm6heDsrR41P/Rx1lXG8aSE8sB60tszvFnM9HoWTEbJTtZo0dNae0A70O27tSrswWhmwKIILeRk9Q0v+3aSJJM0YaDmedgavLNFXrqb7VOqBV78l6XPgMfO8pv2VUO2aLwXWvWiaEM2a7qJpEa9HnF3eDPD8kqZc9eSNcQSALdiSNFSk/NBbdp490DRbNG1bG1hN+vCk46vew/cJxTAkggjptoR0jnF2jhbeOlb3krP/xm4vr24W1Mx9XGcga+Ydj2wpynL0js= 2021-12-18T18:01:10.366 INFO:teuthology.kill: smithi100.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdn/nNOsIR4ERUnDrZnqmHNZ18Ed/LKRt9sGvtTbjaAoaCi3SDSu1y2VAazNTpgCyRZCPj+VCzBMOx0JyZ/UToWAqNopXdwpYcfyI01mtfpFKDk5XDD2Tg4Ov37tF+gzfoXlEfM5JCjowJFm2DyE1xpwGYRRxYtI8JQFbsAFqc1d6/QoVoB0hcRKyMASF7JZuW3eD7ETNYnWuZ0ee6FuXXdxol5cOG3ZLLMI+YuSB6qgulGe+alUKsUCtRYKIat4lN9tI2ebVvplJ18CYuae/3AgRv7nuXqqNJo+tT5qL+qYzUcCbkQphFUMTa39ttCDl8sQHaxYR6O63v1WIBCFJ8Y6U2tw5woiKaTtFhd0FeDpSvqq2U8f+d3D6VwP6xqZYPZddpkwO2QPdhzI+kWbrfdFkXU35W/MajDrMqspUn0NSSkegK0zFYOOGGvYmcSw4QSZ0SONg4Lvh72rPJCp39gE9UDvNeTBAXe2eq9ubNPu1grzCXcv9g1y770G0KSrM= 2021-12-18T18:01:10.391 INFO:teuthology.kill:2021-12-18 18:01:10,391.391 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-18T18:01:10.394 INFO:teuthology.kill:2021-12-18 18:01:10,394.394 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-18T18:01:10.469 INFO:teuthology.kill:2021-12-18 18:01:10,469.469 INFO:teuthology.orchestra.console:Power off smithi100 2021-12-18T18:01:10.483 INFO:teuthology.kill:2021-12-18 18:01:10,482.482 INFO:teuthology.orchestra.console:Power off smithi085 2021-12-18T18:01:18.846 INFO:teuthology.kill:2021-12-18 18:01:18,846.846 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-12-18T18:01:18.999 INFO:teuthology.kill:2021-12-18 18:01:18,999.999 INFO:teuthology.lock.ops:unlocked: smithi085.front.sepia.ceph.com 2021-12-18T18:01:22.953 INFO:teuthology.kill:2021-12-18 18:01:22,953.953 INFO:teuthology.orchestra.console:Power off for smithi100 completed 2021-12-18T18:01:23.099 INFO:teuthology.kill:2021-12-18 18:01:23,099.099 INFO:teuthology.lock.ops:unlocked: smithi100.front.sepia.ceph.com 2021-12-18T18:03:23.378 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15