2023-04-06T03:54:06.763 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-06T03:54:06.896 INFO:teuthology.lock.ops:Start node 'smithi135.front.sepia.ceph.com' reimaging 2023-04-06T03:54:06.897 INFO:teuthology.lock.ops:Updating [smithi135.front.sepia.ceph.com]: reset os type and version on server 2023-04-06T03:54:06.897 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2023-04-06T03:54:06.916 INFO:teuthology.lock.ops:Node 'smithi135.front.sepia.ceph.com' reimaging is complete 2023-04-06T03:54:06.916 INFO:teuthology.lock.ops:Start node 'smithi196.front.sepia.ceph.com' reimaging 2023-04-06T03:54:06.917 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: reset os type and version on server 2023-04-06T03:54:06.917 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2023-04-06T03:54:06.941 INFO:teuthology.lock.ops:Node 'smithi196.front.sepia.ceph.com' reimaging is complete 2023-04-06T03:54:07.080 INFO:teuthology.provision.fog.smithi135:Scheduling deploy of centos 8.stream 2023-04-06T03:54:07.099 INFO:teuthology.provision.fog.smithi196:Scheduling deploy of centos 8.stream 2023-04-06T03:54:07.290 INFO:teuthology.orchestra.console:Power off smithi196 2023-04-06T03:54:07.308 INFO:teuthology.orchestra.console:Power off smithi135 2023-04-06T03:54:15.668 INFO:teuthology.orchestra.console:Power off for smithi196 completed 2023-04-06T03:54:15.680 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2023-04-06T03:54:15.769 INFO:teuthology.orchestra.console:Power on smithi196 2023-04-06T03:54:15.780 INFO:teuthology.orchestra.console:Power on smithi135 2023-04-06T03:54:24.137 INFO:teuthology.orchestra.console:Power on for smithi196 completed 2023-04-06T03:54:24.229 INFO:teuthology.orchestra.console:Power on for smithi135 completed 2023-04-06T03:54:24.238 INFO:teuthology.provision.fog.smithi196:Waiting for deploy to finish 2023-04-06T03:54:24.330 INFO:teuthology.provision.fog.smithi135:Waiting for deploy to finish 2023-04-06T03:57:13.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.196 2023-04-06T03:57:13.061 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-04-06T03:57:22.085 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.196 2023-04-06T03:57:22.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-04-06T03:57:31.173 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.196 2023-04-06T03:57:31.174 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-04-06T03:57:40.233 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.196 2023-04-06T03:57:40.233 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-04-06T03:57:49.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.196 2023-04-06T03:57:49.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-04-06T03:57:58.377 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-04-06T03:57:58.378 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.196 2023-04-06T03:59:04.441 WARNING:teuthology.provision.fog:timed out 2023-04-06T03:59:04.442 WARNING:teuthology.provision.fog:timed out 2023-04-06T03:59:10.730 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-04-06T03:59:10.792 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-04-06T03:59:10.999 INFO:teuthology.provision.fog.smithi135:Node is ready 2023-04-06T03:59:11.056 INFO:teuthology.orchestra.run.smithi135.stdout:smithi135.front.sepia.ceph.com 2023-04-06T03:59:11.058 INFO:teuthology.provision.fog.smithi196:Node is ready 2023-04-06T03:59:11.115 INFO:teuthology.orchestra.run.smithi135.stdout:172.21.15.135 smithi135.front.sepia.ceph.com smithi135 2023-04-06T03:59:11.116 INFO:teuthology.orchestra.run.smithi196.stdout:smithi196.front.sepia.ceph.com 2023-04-06T03:59:11.173 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.15.196 smithi196.front.sepia.ceph.com smithi196 2023-04-06T03:59:11.468 INFO:teuthology.provision.fog.smithi135:Deploy complete! 2023-04-06T03:59:11.523 INFO:teuthology.provision.fog.smithi196:Deploy complete! 2023-04-06T03:59:11.696 INFO:teuthology.lock.ops:Checking smithi135.front.sepia.ceph.com 2023-04-06T03:59:11.697 INFO:teuthology.lock.ops:New key found. Updating... 2023-04-06T03:59:11.720 INFO:teuthology.lock.ops:Checking smithi196.front.sepia.ceph.com 2023-04-06T03:59:11.721 INFO:teuthology.lock.ops:New key found. Updating... 2023-04-06T03:59:11.746 INFO:teuthology.lock.ops:Updating [smithi135.front.sepia.ceph.com]: set os type and version on server 2023-04-06T03:59:11.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-04-06T03:59:11.934 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2023-04-06T03:59:11.991 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2023-04-06T03:59:11.992 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="8" 2023-04-06T03:59:11.992 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2023-04-06T03:59:11.992 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2023-04-06T03:59:11.992 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="8" 2023-04-06T03:59:11.992 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el8" 2023-04-06T03:59:11.993 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 8" 2023-04-06T03:59:11.993 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2023-04-06T03:59:11.993 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-04-06T03:59:11.993 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2023-04-06T03:59:11.993 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-04-06T03:59:11.993 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-04-06T03:59:11.993 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-04-06T03:59:11.994 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2023-04-06T03:59:12.020 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: set os type and version on server 2023-04-06T03:59:12.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2023-04-06T03:59:12.207 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2023-04-06T03:59:12.264 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2023-04-06T03:59:12.264 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="8" 2023-04-06T03:59:12.264 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2023-04-06T03:59:12.264 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2023-04-06T03:59:12.264 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="8" 2023-04-06T03:59:12.265 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el8" 2023-04-06T03:59:12.265 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 8" 2023-04-06T03:59:12.265 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2023-04-06T03:59:12.265 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-04-06T03:59:12.265 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2023-04-06T03:59:12.265 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-04-06T03:59:12.265 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-04-06T03:59:12.266 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-04-06T03:59:12.266 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2023-04-06T03:59:12.382 INFO:teuthology.dispatcher.supervisor:Running job 7233263 2023-04-06T03:59:12.382 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-04-05_20:32:43-rados-wip-yuri8-testing-2023-03-31-0812-pacific-distro-default-smithi/7233263 --name yuriw-2023-04-05_20:32:43-rados-wip-yuri8-testing-2023-03-31-0812-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-04-05_20:32:43-rados-wip-yuri8-testing-2023-03-31-0812-pacific-distro-default-smithi/7233263/orig.config.yaml 2023-04-06T03:59:12.385 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-04-05_20:32:43-rados-wip-yuri8-testing-2023-03-31-0812-pacific-distro-default-smithi/7233263 2023-04-06T03:59:12.386 INFO:teuthology.dispatcher.supervisor:Job PID: 11154 2023-04-06T03:59:12.386 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-04-06T16:00:09.787 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-04-06T16:00:10.046 INFO:teuthology.kill:Killing Pids: {11154} 2023-04-06T16:00:10.930 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-04-06T16:00:10.931 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-04-06T16:00:10.931 INFO:teuthology.misc:Compressing logs... 2023-04-06T16:00:10.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-04-06T16:00:11.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2023-04-06T16:00:11.419 INFO:teuthology.misc:Compressing logs... 2023-04-06T16:00:11.454 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2023-04-06T16:00:11.454 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2023-04-06T16:00:11.455 INFO:teuthology.orchestra.run.smithi135.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 86.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-04-06T16:00:11.489 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2023-04-06T16:00:11.489 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2023-04-06T16:00:11.490 INFO:teuthology.orchestra.run.smithi196.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 77.8% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-04-06T16:00:11.565 INFO:teuthology.orchestra.run.smithi135.stderr: 93.3% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-04-06T16:00:11.572 INFO:teuthology.orchestra.run.smithi196.stderr: 91.7% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-04-06T16:00:11.690 INFO:teuthology.misc:Compressing logs... 2023-04-06T16:00:11.717 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-volume.log 2023-04-06T16:00:11.718 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi135.zmejjw.log 2023-04-06T16:00:11.718 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.2.log 2023-04-06T16:00:11.718 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-volume.log: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi135.zmejjw.log: gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.1.log 2023-04-06T16:00:11.719 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi135.jqkjdh.log 2023-04-06T16:00:11.719 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mon.smithi135.log 2023-04-06T16:00:11.722 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi135.jqkjdh.log: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mon.smithi135.log: 89.7% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-volume.log.gz 2023-04-06T16:00:11.722 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mgr.smithi135.tqowfy.log 2023-04-06T16:00:11.723 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.0.log 2023-04-06T16:00:11.724 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mgr.smithi135.tqowfy.log: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-04-06T16:00:11.730 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/cephadm.log: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2023-04-06T16:00:11.756 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-volume.log 2023-04-06T16:00:11.757 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi196.xyuubl.log 2023-04-06T16:00:11.757 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.5.log 2023-04-06T16:00:11.758 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mgr.smithi196.edwdlv.log 2023-04-06T16:00:11.758 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi196.xyuubl.log: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mon.smithi196.log 2023-04-06T16:00:11.758 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mgr.smithi196.edwdlv.log: gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi196.dgfisq.log 2023-04-06T16:00:11.760 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mon.smithi196.log: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi196.dgfisq.log: gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.3.log 2023-04-06T16:00:11.761 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.3.log: 89.0% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-volume.log.gz 2023-04-06T16:00:11.761 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.4.log 2023-04-06T16:00:11.766 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-04-06T16:00:11.773 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/cephadm.log: 88.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-04-06T16:00:11.835 INFO:teuthology.orchestra.run.smithi196.stderr: 95.1% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi196.xyuubl.log.gz 2023-04-06T16:00:11.837 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi135.zmejjw.log: file size changed while zipping 2023-04-06T16:00:11.837 INFO:teuthology.orchestra.run.smithi135.stderr: 95.1% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi135.zmejjw.log.gz 2023-04-06T16:00:11.913 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mgr.smithi196.edwdlv.log.gz 2023-04-06T16:00:14.055 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mgr.smithi135.tqowfy.log: file size changed while zipping 2023-04-06T16:00:14.056 INFO:teuthology.orchestra.run.smithi135.stderr: 92.4% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mgr.smithi135.tqowfy.log.gz 2023-04-06T16:00:16.204 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mon.smithi196.log: file size changed while zipping 2023-04-06T16:00:16.204 INFO:teuthology.orchestra.run.smithi196.stderr: 93.6% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mon.smithi196.log.gz 2023-04-06T16:00:23.259 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2023-04-06T16:00:23.260 INFO:teuthology.orchestra.run.smithi196.stderr: 93.5% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.4.log.gz 2023-04-06T16:00:23.306 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2023-04-06T16:00:23.306 INFO:teuthology.orchestra.run.smithi135.stderr: 93.5% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.2.log.gz 2023-04-06T16:00:26.204 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2023-04-06T16:00:26.205 INFO:teuthology.orchestra.run.smithi196.stderr: 93.6% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.5.log.gz 2023-04-06T16:00:26.380 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mon.smithi135.log: file size changed while zipping 2023-04-06T16:00:26.380 INFO:teuthology.orchestra.run.smithi135.stderr: 89.9% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mon.smithi135.log.gz 2023-04-06T16:00:27.930 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2023-04-06T16:00:27.930 INFO:teuthology.orchestra.run.smithi135.stderr: 93.7% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.0.log.gz 2023-04-06T16:00:28.407 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2023-04-06T16:00:28.408 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.3.log.gz 2023-04-06T16:00:28.795 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2023-04-06T16:00:28.795 INFO:teuthology.orchestra.run.smithi135.stderr: 93.7% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-osd.1.log.gz 2023-04-06T16:00:43.901 INFO:teuthology.orchestra.run.smithi196.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi196.dgfisq.log: file size changed while zipping 2023-04-06T16:00:43.902 INFO:teuthology.orchestra.run.smithi196.stderr: 93.7% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi196.dgfisq.log.gz 2023-04-06T16:01:04.213 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi135.jqkjdh.log: file size changed while zipping 2023-04-06T16:01:04.213 INFO:teuthology.orchestra.run.smithi135.stderr: 93.5% -- replaced with /var/log/ceph/a34f5792-d43c-11ed-9aff-001a4aab830c/ceph-mds.cephfs.smithi135.jqkjdh.log.gz 2023-04-06T16:01:25.258 INFO:teuthology.kill:No teuthology processes running 2023-04-06T16:01:26.052 INFO:teuthology.kill:Nuking machines: ['smithi135', 'smithi196'] 2023-04-06T16:01:26.640 INFO:teuthology.kill:2023-04-06 16:01:26,640.640 INFO:teuthology.nuke:targets: 2023-04-06T16:01:26.641 INFO:teuthology.kill: smithi135.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCduEjx3o5Slkon4Tk9IR2fg3ZHnt49Jr8TMk2Ra04N3jrWyZ1Fu3EZxoarWlaJiqDY7SRpcD+RBnrf8m9GZa7McXM/wMznHqoDvOjyBVdT0YH5II0XUvN0IbRwsGv4mngH7sNLNFxEqPlYMvHmF0l48ymeJbPoDtYpOFR9L0pFeUgBGzKVFNu93DCVlt6hffPBL4kiYEHIpRyDUx7707Y1cQZdUugSL6qHLK0S2yr+xx431lIUJib/xDm/PNN77mgV26AEl4kllpKAHkkmlU2ePCIc3E0eeCOE/cURqDgJi6x1HQTzIcN4qa1RbxW9ZcOUAF/GWvNX4627esSt50TxzhZxWbV8kEu62xGadGn6ePmjMIGYBB09VqDn5zgXeKhgkGsnF0AZDCd4XTmLqURTbsE5ho/TQGT+ZDquXt6nFJAYefGfNoZE06ib7SIGk0LSwlyZ90PSWh64c8w7qHW2p6NEKS3iw3zII/S49URplDuHgPcxYvJr1Hu1yP8fVTU= 2023-04-06T16:01:26.642 INFO:teuthology.kill: smithi196.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWVKXSeTSH0WsMj+YvskXfO884RZoCaKutgJ2h38nn6Kz5xj0rn1jsqLjnJaWsY5/5SnjjMNCF3mdxrvSOrAG0gIP6K+XERIwK5Wjyg3NjyUOM7+iJX+mdALEs+/LP61mCHL8UR9zFKkIOilUH1oTe0QPhGkVk4tRGEPcFpZf6VogqzXXCdb7J8wV0PBtTlIKbVp7pqE11UTOxwLcWu99KYpeWuOjXJUafRFbQRjp5Wg2mBrldXjlxm50IeAPf9bViRbzzmruJwvfSxLdpkk+XY6C7Lu1xQk96VcrZh6NVfNB7vCJ+YxTmWK1ujaOZQfrrlmHbtApFrPRINRyzQEA/JxwTWY9p5rzKFOorQnfqJRAsnH/c++qxiLCtKfSHy7876B3yNZCS5aGqDRdKh6xTpwNuzLYDzUPLdvuFqz/RBOZOaqjrzBZZySrLGgjp9ytW6tzkcuSZJkP7Zo0dwjjNz3cwpBCafZn6IJ5qE/zmGRdWhA9NsoxWsrxbinhprk8= 2023-04-06T16:01:26.642 INFO:teuthology.kill:2023-04-06 16:01:26,640.640 INFO:teuthology.nuke:Checking targets against current locks 2023-04-06T16:01:26.698 INFO:teuthology.kill:2023-04-06 16:01:26,698.698 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-04-06T16:01:26.722 INFO:teuthology.kill:2023-04-06 16:01:26,722.722 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-04-06T16:01:26.779 INFO:teuthology.kill:2023-04-06 16:01:26,779.779 INFO:teuthology.orchestra.console:Power off smithi135 2023-04-06T16:01:26.823 INFO:teuthology.kill:2023-04-06 16:01:26,822.822 INFO:teuthology.orchestra.console:Power off smithi196 2023-04-06T16:01:39.270 INFO:teuthology.kill:2023-04-06 16:01:39,269.269 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2023-04-06T16:01:39.298 INFO:teuthology.kill:2023-04-06 16:01:39,298.298 INFO:teuthology.orchestra.console:Power off for smithi196 completed 2023-04-06T16:01:39.434 INFO:teuthology.kill:2023-04-06 16:01:39,433.433 INFO:teuthology.lock.ops:unlocked: smithi135.front.sepia.ceph.com 2023-04-06T16:01:39.450 INFO:teuthology.kill:2023-04-06 16:01:39,450.450 INFO:teuthology.lock.ops:unlocked: smithi196.front.sepia.ceph.com 2023-04-06T16:03:39.810 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-04-06T16:03:39.926 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi135 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-04-06_15:37:58-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi/7234295 2023-04-06T16:03:39.926 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi196 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-04-06_15:37:58-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi/7234295