2022-02-27T07:03:40.951 INFO:root:teuthology version: 1.1.0-5825e8d0
2022-02-27T07:03:41.099 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi133.front.sepia.ceph.com' reimaging
2022-02-27T07:03:41.100 INFO:teuthology.lock.ops:Updating [ubuntu@smithi133.front.sepia.ceph.com]: reset os type and version on server
2022-02-27T07:03:41.100 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server
2022-02-27T07:03:41.119 INFO:teuthology.lock.ops:Node 'ubuntu@smithi133.front.sepia.ceph.com' reimaging is complete
2022-02-27T07:03:41.120 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi173.front.sepia.ceph.com' reimaging
2022-02-27T07:03:41.120 INFO:teuthology.lock.ops:Updating [ubuntu@smithi173.front.sepia.ceph.com]: reset os type and version on server
2022-02-27T07:03:41.120 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server
2022-02-27T07:03:41.141 INFO:teuthology.lock.ops:Node 'ubuntu@smithi173.front.sepia.ceph.com' reimaging is complete
2022-02-27T07:03:41.344 INFO:teuthology.provision.fog.smithi133:Scheduling deploy of centos 8.stream
2022-02-27T07:03:41.350 INFO:teuthology.provision.fog.smithi173:Scheduling deploy of centos 8.stream
2022-02-27T07:03:41.667 INFO:teuthology.orchestra.console:Power off smithi173
2022-02-27T07:03:41.707 INFO:teuthology.orchestra.console:Power off smithi133
2022-02-27T07:03:50.093 INFO:teuthology.orchestra.console:Power off for smithi173 completed
2022-02-27T07:03:50.108 INFO:teuthology.orchestra.console:Power off for smithi133 completed
2022-02-27T07:03:50.195 INFO:teuthology.orchestra.console:Power on smithi173
2022-02-27T07:03:50.210 INFO:teuthology.orchestra.console:Power on smithi133
2022-02-27T07:03:58.573 INFO:teuthology.orchestra.console:Power on for smithi173 completed
2022-02-27T07:03:58.647 INFO:teuthology.orchestra.console:Power on for smithi133 completed
2022-02-27T07:03:58.675 INFO:teuthology.provision.fog.smithi173:Waiting for deploy to finish
2022-02-27T07:03:58.749 INFO:teuthology.provision.fog.smithi133:Waiting for deploy to finish
2022-02-27T07:08:39.581 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-27T07:08:39.850 INFO:teuthology.provision.fog.smithi133:Node is ready
2022-02-27T07:08:39.871 INFO:teuthology.orchestra.run.smithi133.stdout:smithi133.front.sepia.ceph.com
2022-02-27T07:08:39.930 INFO:teuthology.orchestra.run.smithi133.stdout:172.21.15.133 smithi133.front.sepia.ceph.com smithi133
2022-02-27T07:08:40.289 INFO:teuthology.provision.fog.smithi133:Deploy complete!
2022-02-27T07:08:50.476 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-27T07:09:15.667 INFO:teuthology.provision.fog.smithi173:Node is ready
2022-02-27T07:09:15.694 INFO:teuthology.orchestra.run.smithi173.stdout:smithi173.front.sepia.ceph.com
2022-02-27T07:09:15.752 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.15.173 smithi173.front.sepia.ceph.com smithi173
2022-02-27T07:09:16.119 INFO:teuthology.provision.fog.smithi173:Deploy complete!
2022-02-27T07:09:16.289 INFO:teuthology.lock.ops:Checking smithi133.front.sepia.ceph.com
2022-02-27T07:09:16.290 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-27T07:09:16.314 INFO:teuthology.lock.ops:Checking smithi173.front.sepia.ceph.com
2022-02-27T07:09:16.315 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-27T07:09:16.336 INFO:teuthology.lock.ops:Updating [smithi133.front.sepia.ceph.com]: set os type and version on server
2022-02-27T07:09:16.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com'
2022-02-27T07:09:16.587 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64
2022-02-27T07:09:16.644 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="CentOS Stream"
2022-02-27T07:09:16.645 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="8"
2022-02-27T07:09:16.645 INFO:teuthology.orchestra.run.smithi133.stdout:ID="centos"
2022-02-27T07:09:16.645 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="rhel fedora"
2022-02-27T07:09:16.646 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="8"
2022-02-27T07:09:16.646 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el8"
2022-02-27T07:09:16.646 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="CentOS Stream 8"
2022-02-27T07:09:16.646 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31"
2022-02-27T07:09:16.647 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-02-27T07:09:16.647 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://centos.org/"
2022-02-27T07:09:16.647 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-27T07:09:16.647 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-27T07:09:16.648 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-02-27T07:09:16.649 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server
2022-02-27T07:09:16.669 INFO:teuthology.lock.ops:Updating [smithi173.front.sepia.ceph.com]: set os type and version on server
2022-02-27T07:09:16.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com'
2022-02-27T07:09:16.863 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64
2022-02-27T07:09:16.920 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream"
2022-02-27T07:09:16.921 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="8"
2022-02-27T07:09:16.921 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos"
2022-02-27T07:09:16.921 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora"
2022-02-27T07:09:16.922 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="8"
2022-02-27T07:09:16.922 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el8"
2022-02-27T07:09:16.923 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 8"
2022-02-27T07:09:16.923 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31"
2022-02-27T07:09:16.924 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-02-27T07:09:16.924 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/"
2022-02-27T07:09:16.924 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-27T07:09:16.924 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-27T07:09:16.925 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-02-27T07:09:16.925 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server
2022-02-27T07:09:17.036 INFO:teuthology.dispatcher.supervisor:Running job 6708636
2022-02-27T07:09:17.037 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_5825e8d094b6aff4351d26f1ceb5efc598c156a0/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-02-27_03:31:03-rados-pacific-distro-default-smithi/6708636 --name teuthology-2022-02-27_03:31:03-rados-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/teuthology-2022-02-27_03:31:03-rados-pacific-distro-default-smithi/6708636/orig.config.yaml
2022-02-27T07:09:17.046 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-02-27_03:31:03-rados-pacific-distro-default-smithi/6708636
2022-02-27T07:09:17.048 INFO:teuthology.dispatcher.supervisor:Job PID: 9484
2022-02-27T07:09:17.049 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-27T13:39:46.159 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-27T13:39:46.398 INFO:teuthology.kill:Killing Pids: {9484}
2022-02-27T13:39:47.639 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2022-02-27T13:39:47.640 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2022-02-27T13:39:47.641 INFO:teuthology.misc:Compressing logs...
2022-02-27T13:39:47.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com'
2022-02-27T13:39:47.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com'
2022-02-27T13:39:48.153 INFO:teuthology.misc:Compressing logs...
2022-02-27T13:39:48.851 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-02-27T13:39:49.120 INFO:teuthology.misc:Compressing logs...
2022-02-27T13:39:51.176 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-mgr.smithi133.qtgcga.log: file size changed while zipping
2022-02-27T13:39:51.338 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi173.fwuwnj.log: file size changed while zipping
2022-02-27T13:39:51.421 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-mgr.smithi173.cxuqhi.log: file size changed while zipping
2022-02-27T13:40:03.854 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-02-27T13:40:09.048 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-02-27T13:40:12.633 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-mon.smithi173.log: file size changed while zipping
2022-02-27T13:40:13.378 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-02-27T13:40:16.814 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi173.aescws.log: file size changed while zipping
2022-02-27T13:40:27.136 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-02-27T13:40:36.970 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi133.ykstrl.log: file size changed while zipping
2022-02-27T13:40:48.219 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-mon.smithi133.log: file size changed while zipping
2022-02-27T13:40:59.401 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi133.volioi.log: file size changed while zipping
2022-02-27T13:41:10.888 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/a9ba2382-979d-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-02-27T13:41:37.025 INFO:teuthology.kill:No teuthology processes running
2022-02-27T13:41:38.231 INFO:teuthology.kill:Nuking machines: ['smithi133', 'smithi173']
2022-02-27T13:41:39.243 INFO:teuthology.kill:2022-02-27 13:41:39,242.242 INFO:teuthology.nuke:targets:
2022-02-27T13:41:39.244 INFO:teuthology.kill:  smithi133.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFEmDSpztGwdld8oBMMLERM07R+y0JnR3GUfK5KVkor80D66c/at7l/3jelSeFEW0vnqdz8VQwfXUq2pMm/kTE+HMCUG6F/5tzeGxQX+rAeXIinp0BrvmOl1deyzkabJ40mFbPJMm8PcV6077KGpjoKF3cbN5de4KmEU/TP1dqjy5hcR86MKJSsZQlhrAE+al8KiqF30mLMurUDmGt1byPYF4lJNfUerB/+DprKaQpDraeEHPuLbIVGhluCp7FtrXl5rPstESPsCUfFb43Ryf9Dt51LcTfyyf+Bylb/Zxj6N4eGSyMrdby26VAhEzROrHDboODCe9YYtKAnpCJUi4MEDwDcdSoEawlj3OyfYzuayH/rPQLU8603MxohomdJgMI8A1DKwUmhm7p02qxxquKLp2Gc/W1y6RvqLxPBt0147AqsKqOt6RMi5JZ7zdCHYQjWyi5QrizUsoK/R0j2Is8yC2N8gQ8kx/QG9VVC8NSS8OxB6dYwnJtY8iVfSFE/mk=
2022-02-27T13:41:39.244 INFO:teuthology.kill:  smithi173.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs5PTeycVQ4UhodfyXNtwjZRqbjG536tnL0u9ezZWy6yT6ElMCQeQZg0oHj6I2n9nBGlZ2p+No6zU898Yv2J7kqwNBllJU5FVLT2qOUanpP0B/gaGIrI2zyB1YITP5XRPH1+a4X1AP3v0MEU5upPqLmjKs/nSAXC4VYBE9Ks0U7GbWYB1reZk88V6Ujk0vBkq5ER4HSPiQWOgsP9XtmLcBpYq0MVwNUgqzjWSHBmBD8F7ogcdMQ61koYkhjSZfYcK2B3P7COXmE7f09yawgN1Wn1DfPIqgnQyZmMDWpgFE4L2Ftsacv1+7nwNe5UvFp7GwdYFlO9NOkqi0ojJATmQ1L1XhfvaXC2UxtnxjZxAkyqR1F2YE9c6N4RLLn/B+/oRWfpcNh5UNaszZQV/dOUdnc+jCoRr4g6By95BpbVxFiGrURqf9Ut/ud8QNHNKPdFSE8Dkb2YLrt3FxEkxwSvTnW8EI6ryU+ClKH/dwep9mLfTbVQYzRGhEToha0Lf3tRc=
2022-02-27T13:41:39.270 INFO:teuthology.kill:2022-02-27 13:41:39,270.270 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-27T13:41:39.274 INFO:teuthology.kill:2022-02-27 13:41:39,273.273 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-27T13:41:39.348 INFO:teuthology.kill:2022-02-27 13:41:39,348.348 INFO:teuthology.orchestra.console:Power off smithi173
2022-02-27T13:41:39.365 INFO:teuthology.kill:2022-02-27 13:41:39,365.365 INFO:teuthology.orchestra.console:Power off smithi133
2022-02-27T13:41:47.721 INFO:teuthology.kill:2022-02-27 13:41:47,720.720 INFO:teuthology.orchestra.console:Power off for smithi173 completed
2022-02-27T13:41:47.726 INFO:teuthology.kill:2022-02-27 13:41:47,726.726 INFO:teuthology.orchestra.console:Power off for smithi133 completed
2022-02-27T13:41:47.872 INFO:teuthology.kill:2022-02-27 13:41:47,871.871 INFO:teuthology.lock.ops:unlocked: smithi173.front.sepia.ceph.com
2022-02-27T13:41:47.885 INFO:teuthology.kill:2022-02-27 13:41:47,884.884 INFO:teuthology.lock.ops:unlocked: smithi133.front.sepia.ceph.com
2022-02-27T13:43:48.257 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-27T13:43:48.395 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi133 but it was locked by another job: /home/teuthworker/archive/ozeneva-2022-02-27_11:02:05-rados-wip-omri-tracing-compiled-distro-basic-smithi/6709654
2022-02-27T13:43:48.395 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi173 but it was locked by another job: /home/teuthworker/archive/ozeneva-2022-02-27_11:02:05-rados-wip-omri-tracing-compiled-distro-basic-smithi/6709654