2023-03-10T03:35:35.139 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05
2023-03-10T03:35:35.322 INFO:teuthology.lock.ops:Start node 'smithi008.front.sepia.ceph.com' reimaging
2023-03-10T03:35:35.323 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: reset os type and version on server
2023-03-10T03:35:35.323 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server
2023-03-10T03:35:35.349 INFO:teuthology.lock.ops:Node 'smithi008.front.sepia.ceph.com' reimaging is complete
2023-03-10T03:35:35.349 INFO:teuthology.lock.ops:Start node 'smithi146.front.sepia.ceph.com' reimaging
2023-03-10T03:35:35.350 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: reset os type and version on server
2023-03-10T03:35:35.350 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server
2023-03-10T03:35:35.374 INFO:teuthology.lock.ops:Node 'smithi146.front.sepia.ceph.com' reimaging is complete
2023-03-10T03:35:35.374 INFO:teuthology.lock.ops:Start node 'smithi193.front.sepia.ceph.com' reimaging
2023-03-10T03:35:35.374 INFO:teuthology.lock.ops:Updating [smithi193.front.sepia.ceph.com]: reset os type and version on server
2023-03-10T03:35:35.374 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server
2023-03-10T03:35:35.398 INFO:teuthology.lock.ops:Node 'smithi193.front.sepia.ceph.com' reimaging is complete
2023-03-10T03:35:35.508 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of rhel 8.6
2023-03-10T03:35:35.520 INFO:teuthology.provision.fog.smithi146:Scheduling deploy of rhel 8.6
2023-03-10T03:35:35.552 INFO:teuthology.provision.fog.smithi193:Scheduling deploy of rhel 8.6
2023-03-10T03:35:35.734 INFO:teuthology.orchestra.console:Power off smithi008
2023-03-10T03:35:35.755 INFO:teuthology.orchestra.console:Power off smithi146
2023-03-10T03:35:35.780 INFO:teuthology.orchestra.console:Power off smithi193
2023-03-10T03:35:44.128 INFO:teuthology.orchestra.console:Power off for smithi008 completed
2023-03-10T03:35:44.138 INFO:teuthology.orchestra.console:Power off for smithi146 completed
2023-03-10T03:35:44.230 INFO:teuthology.orchestra.console:Power on smithi008
2023-03-10T03:35:44.239 INFO:teuthology.orchestra.console:Power on smithi146
2023-03-10T03:35:48.259 INFO:teuthology.orchestra.console:Power off for smithi193 completed
2023-03-10T03:35:48.360 INFO:teuthology.orchestra.console:Power on smithi193
2023-03-10T03:35:52.603 INFO:teuthology.orchestra.console:Power on for smithi146 completed
2023-03-10T03:35:52.614 INFO:teuthology.orchestra.console:Power on for smithi193 completed
2023-03-10T03:35:52.658 INFO:teuthology.orchestra.console:Power on for smithi008 completed
2023-03-10T03:35:52.704 INFO:teuthology.provision.fog.smithi146:Waiting for deploy to finish
2023-03-10T03:35:52.715 INFO:teuthology.provision.fog.smithi193:Waiting for deploy to finish
2023-03-10T03:35:52.760 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish
2023-03-10T03:38:42.053 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.8
2023-03-10T03:38:51.141 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.8
2023-03-10T03:38:56.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-03-10T03:38:56.902 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193
2023-03-10T03:39:00.197 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.8
2023-03-10T03:39:05.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-03-10T03:39:05.958 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193
2023-03-10T03:39:09.253 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.8
2023-03-10T03:39:15.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-03-10T03:39:15.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193
2023-03-10T03:39:24.073 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-03-10T03:39:24.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193
2023-03-10T03:39:33.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-03-10T03:39:42.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-03-10T03:40:15.281 WARNING:teuthology.provision.fog:timed out
2023-03-10T03:40:20.112 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-10T03:40:20.406 INFO:teuthology.provision.fog.smithi146:Node is ready
2023-03-10T03:40:20.422 INFO:teuthology.orchestra.run.smithi146.stdout:smithi022
2023-03-10T03:40:20.479 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022
2023-03-10T03:40:20.805 INFO:teuthology.provision.fog.smithi146:Deploy complete!
2023-03-10T03:40:21.728 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-10T03:40:21.998 INFO:teuthology.provision.fog.smithi008:Node is ready
2023-03-10T03:40:22.015 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008.front.sepia.ceph.com
2023-03-10T03:40:22.074 INFO:teuthology.orchestra.run.smithi008.stdout:172.21.15.8 smithi008.front.sepia.ceph.com smithi008
2023-03-10T03:40:22.429 INFO:teuthology.provision.fog.smithi008:Deploy complete!
2023-03-10T03:40:30.109 WARNING:teuthology.provision.fog:timed out
2023-03-10T03:40:36.480 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-10T03:40:36.749 INFO:teuthology.provision.fog.smithi193:Node is ready
2023-03-10T03:40:36.768 INFO:teuthology.orchestra.run.smithi193.stdout:smithi193
2023-03-10T03:40:36.827 INFO:teuthology.orchestra.run.smithi193.stdout:172.21.15.193 smithi193.front.sepia.ceph.com smithi193
2023-03-10T03:40:37.185 INFO:teuthology.provision.fog.smithi193:Deploy complete!
2023-03-10T03:40:37.386 INFO:teuthology.lock.ops:Checking smithi008.front.sepia.ceph.com
2023-03-10T03:40:37.386 INFO:teuthology.lock.ops:Checking smithi146.front.sepia.ceph.com
2023-03-10T03:40:37.387 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-10T03:40:37.412 INFO:teuthology.lock.ops:Checking smithi193.front.sepia.ceph.com
2023-03-10T03:40:37.413 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: set os type and version on server
2023-03-10T03:40:37.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com'
2023-03-10T03:40:37.673 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64
2023-03-10T03:40:37.731 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="Red Hat Enterprise Linux"
2023-03-10T03:40:37.731 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="8.6 (Ootpa)"
2023-03-10T03:40:37.731 INFO:teuthology.orchestra.run.smithi008.stdout:ID="rhel"
2023-03-10T03:40:37.732 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="fedora"
2023-03-10T03:40:37.732 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="8.6"
2023-03-10T03:40:37.732 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el8"
2023-03-10T03:40:37.732 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-10T03:40:37.732 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31"
2023-03-10T03:40:37.732 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-10T03:40:37.732 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://www.redhat.com/"
2023-03-10T03:40:37.732 INFO:teuthology.orchestra.run.smithi008.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-10T03:40:37.733 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-10T03:40:37.733 INFO:teuthology.orchestra.run.smithi008.stdout:
2023-03-10T03:40:37.733 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-10T03:40:37.733 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-10T03:40:37.733 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-10T03:40:37.733 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-10T03:40:37.734 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server
2023-03-10T03:40:37.762 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: set os type and version on server
2023-03-10T03:40:37.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-03-10T03:40:38.019 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64
2023-03-10T03:40:38.077 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Red Hat Enterprise Linux"
2023-03-10T03:40:38.078 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8.6 (Ootpa)"
2023-03-10T03:40:38.078 INFO:teuthology.orchestra.run.smithi146.stdout:ID="rhel"
2023-03-10T03:40:38.078 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="fedora"
2023-03-10T03:40:38.078 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8.6"
2023-03-10T03:40:38.078 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8"
2023-03-10T03:40:38.078 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-10T03:40:38.079 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31"
2023-03-10T03:40:38.079 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-10T03:40:38.079 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.redhat.com/"
2023-03-10T03:40:38.079 INFO:teuthology.orchestra.run.smithi146.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-10T03:40:38.079 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-10T03:40:38.079 INFO:teuthology.orchestra.run.smithi146.stdout:
2023-03-10T03:40:38.079 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-10T03:40:38.080 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-10T03:40:38.080 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-10T03:40:38.080 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-10T03:40:38.080 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server
2023-03-10T03:40:38.103 INFO:teuthology.lock.ops:Updating [smithi193.front.sepia.ceph.com]: set os type and version on server
2023-03-10T03:40:38.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-10T03:40:38.303 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64
2023-03-10T03:40:38.362 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="Red Hat Enterprise Linux"
2023-03-10T03:40:38.363 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8.6 (Ootpa)"
2023-03-10T03:40:38.363 INFO:teuthology.orchestra.run.smithi193.stdout:ID="rhel"
2023-03-10T03:40:38.363 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="fedora"
2023-03-10T03:40:38.364 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8.6"
2023-03-10T03:40:38.364 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8"
2023-03-10T03:40:38.364 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-10T03:40:38.365 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31"
2023-03-10T03:40:38.365 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-10T03:40:38.365 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://www.redhat.com/"
2023-03-10T03:40:38.365 INFO:teuthology.orchestra.run.smithi193.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-10T03:40:38.365 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-10T03:40:38.366 INFO:teuthology.orchestra.run.smithi193.stdout:
2023-03-10T03:40:38.366 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-10T03:40:38.366 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-10T03:40:38.366 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-10T03:40:38.366 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-10T03:40:38.367 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server
2023-03-10T03:40:38.515 INFO:teuthology.dispatcher.supervisor:Running job 7198228
2023-03-10T03:40:38.515 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198228 --name pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} -- /home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198228/orig.config.yaml
2023-03-10T03:40:38.520 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198228
2023-03-10T03:40:38.521 INFO:teuthology.dispatcher.supervisor:Job PID: 13707
2023-03-10T03:40:38.521 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-10T15:41:36.280 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-10T15:41:36.452 INFO:teuthology.kill:Killing Pids: {13707}
2023-03-10T15:41:38.012 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-10T15:41:38.013 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-10T15:41:38.013 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-10T15:41:38.013 INFO:teuthology.misc:Compressing logs...
2023-03-10T15:41:38.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com'
2023-03-10T15:41:38.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-03-10T15:41:38.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-10T15:41:39.102 INFO:teuthology.misc:Compressing logs...
2023-03-10T15:41:40.619 INFO:teuthology.orchestra.run.smithi008.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-10T15:41:41.123 INFO:teuthology.misc:Compressing logs...
2023-03-10T15:43:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-osd.10.log: file size changed while zipping
2023-03-10T15:43:56.246 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-mds.h.log: file size changed while zipping
2023-03-10T15:43:57.874 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-mds.e.log: file size changed while zipping
2023-03-10T15:45:53.475 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2023-03-10T15:47:30.233 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2023-03-10T15:48:34.104 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2023-03-10T15:50:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2023-03-10T15:50:50.021 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-osd.11.log: file size changed while zipping
2023-03-10T15:53:25.101 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2023-03-10T15:53:29.483 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-osd.8.log: file size changed while zipping
2023-03-10T15:53:31.317 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-mds.f.log: file size changed while zipping
2023-03-10T15:53:34.593 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-mds.c.log: file size changed while zipping
2023-03-10T15:53:35.534 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2023-03-10T15:53:38.006 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-mds.i.log: file size changed while zipping
2023-03-10T15:53:45.030 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-mds.b.log: file size changed while zipping
2023-03-10T15:53:50.768 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2023-03-10T15:53:59.914 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/04c2ef20-bef8-11ed-9af2-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2023-03-10T15:59:30.412 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 351, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/orchestra/run.py", line 174, in _raise_for_status
    raise ConnectionLostError(command=self.command,
teuthology.exceptions.ConnectionLostError: SSH connection to smithi008 was lost: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-10T15:59:30.535 INFO:teuthology.kill:No teuthology processes running
2023-03-10T15:59:31.305 INFO:teuthology.kill:Nuking machines: ['smithi008', 'smithi146', 'smithi193']
2023-03-10T15:59:32.144 INFO:teuthology.kill:2023-03-10 15:59:32,143.143 INFO:teuthology.nuke:targets:
2023-03-10T15:59:32.145 INFO:teuthology.kill:  smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-10T15:59:32.146 INFO:teuthology.kill:  smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-10T15:59:32.146 INFO:teuthology.kill:  smithi193.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-10T15:59:32.146 INFO:teuthology.kill:2023-03-10 15:59:32,144.144 INFO:teuthology.nuke:Checking targets against current locks
2023-03-10T15:59:32.204 INFO:teuthology.kill:2023-03-10 15:59:32,204.204 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-10T15:59:32.235 INFO:teuthology.kill:2023-03-10 15:59:32,234.234 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-10T15:59:32.256 INFO:teuthology.kill:2023-03-10 15:59:32,256.256 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-10T15:59:32.291 INFO:teuthology.kill:2023-03-10 15:59:32,291.291 INFO:teuthology.orchestra.console:Power off smithi008
2023-03-10T15:59:32.315 INFO:teuthology.kill:2023-03-10 15:59:32,315.315 INFO:teuthology.orchestra.console:Power off smithi146
2023-03-10T15:59:32.385 INFO:teuthology.kill:2023-03-10 15:59:32,384.384 INFO:teuthology.orchestra.console:Power off smithi193
2023-03-10T15:59:36.564 INFO:teuthology.kill:2023-03-10 15:59:36,563.563 INFO:teuthology.orchestra.console:Power off for smithi008 completed
2023-03-10T15:59:36.575 INFO:teuthology.kill:2023-03-10 15:59:36,575.575 INFO:teuthology.orchestra.console:Power off for smithi146 completed
2023-03-10T15:59:36.729 INFO:teuthology.kill:2023-03-10 15:59:36,729.729 INFO:teuthology.lock.ops:unlocked: smithi146.front.sepia.ceph.com
2023-03-10T15:59:40.753 INFO:teuthology.kill:2023-03-10 15:59:40,752.752 INFO:teuthology.orchestra.console:Power off for smithi193 completed
2023-03-10T16:00:03.883 INFO:teuthology.kill:Traceback (most recent call last):
2023-03-10T16:00:03.883 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/virtualenv/bin/teuthology-nuke", line 8, in <module>
2023-03-10T16:00:03.883 INFO:teuthology.kill:    sys.exit(main())
2023-03-10T16:00:03.884 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/scripts/nuke.py", line 47, in main
2023-03-10T16:00:03.884 INFO:teuthology.kill:    teuthology.nuke.main(args)
2023-03-10T16:00:03.884 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/nuke/__init__.py", line 235, in main
2023-03-10T16:00:03.884 INFO:teuthology.kill:    nuke(ctx, ctx.unlock, ctx.synch_clocks, ctx.noipmi, ctx.keep_logs, not ctx.no_reboot)
2023-03-10T16:00:03.884 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/nuke/__init__.py", line 268, in nuke
2023-03-10T16:00:03.884 INFO:teuthology.kill:    for unnuked in p:
2023-03-10T16:00:03.885 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 98, in __next__
2023-03-10T16:00:03.885 INFO:teuthology.kill:    resurrect_traceback(result)
2023-03-10T16:00:03.885 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 30, in resurrect_traceback
2023-03-10T16:00:03.885 INFO:teuthology.kill:    raise exc.exc_info[1]
2023-03-10T16:00:03.885 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 23, in capture_traceback
2023-03-10T16:00:03.886 INFO:teuthology.kill:    return func(*args, **kwargs)
2023-03-10T16:00:03.886 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/nuke/__init__.py", line 299, in nuke_one
2023-03-10T16:00:03.886 INFO:teuthology.kill:    unlock_one(ctx, list(target.keys())[0], ctx.owner)
2023-03-10T16:00:03.886 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/lock/ops.py", line 209, in unlock_one
2023-03-10T16:00:03.886 INFO:teuthology.kill:    while proceed():
2023-03-10T16:00:03.886 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__
2023-03-10T16:00:03.887 INFO:teuthology.kill:    raise MaxWhileTries(error_msg)
2023-03-10T16:00:03.887 INFO:teuthology.kill:teuthology.exceptions.MaxWhileTries: 'unlock smithi008.front.sepia.ceph.com' reached maximum tries (10) after waiting for 32.5 seconds
2023-03-10T16:02:04.247 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-10T16:02:04.422 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi008 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-03-10_14:59:04-rados-wip-yuri4-testing-2023-03-09-1458-reef-distro-default-smithi/7200927
2023-03-10T16:02:04.423 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi146 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-03-10_14:59:04-rados-wip-yuri4-testing-2023-03-09-1458-reef-distro-default-smithi/7200930
2023-03-10T16:02:04.423 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi193 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-03-10_14:59:04-rados-wip-yuri4-testing-2023-03-09-1458-reef-distro-default-smithi/7200932