2023-03-09T19:48:10.687 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05
2023-03-09T19:48:10.883 INFO:teuthology.lock.ops:Start node 'smithi007.front.sepia.ceph.com' reimaging
2023-03-09T19:48:10.884 INFO:teuthology.lock.ops:Updating [smithi007.front.sepia.ceph.com]: reset os type and version on server
2023-03-09T19:48:10.884 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server
2023-03-09T19:48:10.907 INFO:teuthology.lock.ops:Node 'smithi007.front.sepia.ceph.com' reimaging is complete
2023-03-09T19:48:10.908 INFO:teuthology.lock.ops:Start node 'smithi016.front.sepia.ceph.com' reimaging
2023-03-09T19:48:10.908 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: reset os type and version on server
2023-03-09T19:48:10.908 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server
2023-03-09T19:48:10.932 INFO:teuthology.lock.ops:Node 'smithi016.front.sepia.ceph.com' reimaging is complete
2023-03-09T19:48:10.932 INFO:teuthology.lock.ops:Start node 'smithi156.front.sepia.ceph.com' reimaging
2023-03-09T19:48:10.932 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: reset os type and version on server
2023-03-09T19:48:10.933 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server
2023-03-09T19:48:10.957 INFO:teuthology.lock.ops:Node 'smithi156.front.sepia.ceph.com' reimaging is complete
2023-03-09T19:48:11.084 INFO:teuthology.provision.fog.smithi016:Scheduling deploy of rhel 8.6
2023-03-09T19:48:11.108 INFO:teuthology.provision.fog.smithi007:Scheduling deploy of rhel 8.6
2023-03-09T19:48:11.123 INFO:teuthology.provision.fog.smithi156:Scheduling deploy of rhel 8.6
2023-03-09T19:48:11.296 INFO:teuthology.orchestra.console:Power off smithi016
2023-03-09T19:48:11.317 INFO:teuthology.orchestra.console:Power off smithi007
2023-03-09T19:48:11.357 INFO:teuthology.orchestra.console:Power off smithi156
2023-03-09T19:48:19.691 INFO:teuthology.orchestra.console:Power off for smithi016 completed
2023-03-09T19:48:19.698 INFO:teuthology.orchestra.console:Power off for smithi007 completed
2023-03-09T19:48:19.729 INFO:teuthology.orchestra.console:Power off for smithi156 completed
2023-03-09T19:48:19.792 INFO:teuthology.orchestra.console:Power on smithi016
2023-03-09T19:48:19.802 INFO:teuthology.orchestra.console:Power on smithi007
2023-03-09T19:48:19.831 INFO:teuthology.orchestra.console:Power on smithi156
2023-03-09T19:48:28.169 INFO:teuthology.orchestra.console:Power on for smithi016 completed
2023-03-09T19:48:28.175 INFO:teuthology.orchestra.console:Power on for smithi007 completed
2023-03-09T19:48:28.195 INFO:teuthology.orchestra.console:Power on for smithi156 completed
2023-03-09T19:48:28.270 INFO:teuthology.provision.fog.smithi016:Waiting for deploy to finish
2023-03-09T19:48:28.276 INFO:teuthology.provision.fog.smithi007:Waiting for deploy to finish
2023-03-09T19:48:28.296 INFO:teuthology.provision.fog.smithi156:Waiting for deploy to finish
2023-03-09T19:50:59.997 ERROR:teuthology.orchestra.connection:Error authenticating with smithi007.front.sepia.ceph.com: Authentication failed.
2023-03-09T19:51:17.509 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-03-09T19:51:26.569 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-03-09T19:51:32.617 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T19:51:35.625 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-03-09T19:51:41.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T19:51:44.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-03-09T19:51:50.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T19:51:59.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T19:52:01.002 WARNING:teuthology.provision.fog:timed out
2023-03-09T19:52:08.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T19:52:17.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T19:52:25.353 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.7
2023-03-09T19:52:34.718 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-09T19:52:35.038 INFO:teuthology.provision.fog.smithi007:Node is ready
2023-03-09T19:52:35.056 INFO:teuthology.orchestra.run.smithi007.stdout:smithi022
2023-03-09T19:52:35.114 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022
2023-03-09T19:52:35.461 INFO:teuthology.provision.fog.smithi007:Deploy complete!
2023-03-09T19:52:50.679 WARNING:teuthology.provision.fog:timed out
2023-03-09T19:52:57.012 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-09T19:52:57.336 INFO:teuthology.provision.fog.smithi016:Node is ready
2023-03-09T19:52:57.355 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016.front.sepia.ceph.com
2023-03-09T19:52:57.414 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16 smithi016.front.sepia.ceph.com smithi016
2023-03-09T19:52:57.763 INFO:teuthology.provision.fog.smithi016:Deploy complete!
2023-03-09T19:53:23.896 WARNING:teuthology.provision.fog:timed out
2023-03-09T19:53:30.229 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-09T19:53:30.565 INFO:teuthology.provision.fog.smithi156:Node is ready
2023-03-09T19:53:30.584 INFO:teuthology.orchestra.run.smithi156.stdout:smithi156.front.sepia.ceph.com
2023-03-09T19:53:30.641 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.15.156 smithi156.front.sepia.ceph.com smithi156
2023-03-09T19:53:30.995 INFO:teuthology.provision.fog.smithi156:Deploy complete!
2023-03-09T19:53:31.201 INFO:teuthology.lock.ops:Checking smithi007.front.sepia.ceph.com
2023-03-09T19:53:31.202 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-09T19:53:31.224 INFO:teuthology.lock.ops:Checking smithi016.front.sepia.ceph.com
2023-03-09T19:53:31.225 INFO:teuthology.lock.ops:Checking smithi156.front.sepia.ceph.com
2023-03-09T19:53:31.227 INFO:teuthology.lock.ops:Updating [smithi007.front.sepia.ceph.com]: set os type and version on server
2023-03-09T19:53:31.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com'
2023-03-09T19:53:31.493 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64
2023-03-09T19:53:31.551 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Red Hat Enterprise Linux"
2023-03-09T19:53:31.551 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="8.6 (Ootpa)"
2023-03-09T19:53:31.552 INFO:teuthology.orchestra.run.smithi007.stdout:ID="rhel"
2023-03-09T19:53:31.552 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="fedora"
2023-03-09T19:53:31.552 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="8.6"
2023-03-09T19:53:31.552 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el8"
2023-03-09T19:53:31.552 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-09T19:53:31.552 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31"
2023-03-09T19:53:31.552 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-09T19:53:31.553 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.redhat.com/"
2023-03-09T19:53:31.553 INFO:teuthology.orchestra.run.smithi007.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-09T19:53:31.553 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-09T19:53:31.553 INFO:teuthology.orchestra.run.smithi007.stdout:
2023-03-09T19:53:31.553 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-09T19:53:31.553 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-09T19:53:31.554 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-09T19:53:31.554 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-09T19:53:31.555 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server
2023-03-09T19:53:31.611 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: set os type and version on server
2023-03-09T19:53:31.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com'
2023-03-09T19:53:31.827 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64
2023-03-09T19:53:31.885 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Red Hat Enterprise Linux"
2023-03-09T19:53:31.885 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="8.6 (Ootpa)"
2023-03-09T19:53:31.885 INFO:teuthology.orchestra.run.smithi016.stdout:ID="rhel"
2023-03-09T19:53:31.885 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="fedora"
2023-03-09T19:53:31.885 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="8.6"
2023-03-09T19:53:31.886 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el8"
2023-03-09T19:53:31.886 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-09T19:53:31.886 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31"
2023-03-09T19:53:31.886 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-09T19:53:31.886 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.redhat.com/"
2023-03-09T19:53:31.886 INFO:teuthology.orchestra.run.smithi016.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-09T19:53:31.886 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-09T19:53:31.887 INFO:teuthology.orchestra.run.smithi016.stdout:
2023-03-09T19:53:31.887 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-09T19:53:31.887 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-09T19:53:31.887 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-09T19:53:31.887 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-09T19:53:31.888 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server
2023-03-09T19:53:31.916 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: set os type and version on server
2023-03-09T19:53:31.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com'
2023-03-09T19:53:32.104 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64
2023-03-09T19:53:32.159 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="Red Hat Enterprise Linux"
2023-03-09T19:53:32.160 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="8.6 (Ootpa)"
2023-03-09T19:53:32.160 INFO:teuthology.orchestra.run.smithi156.stdout:ID="rhel"
2023-03-09T19:53:32.160 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="fedora"
2023-03-09T19:53:32.160 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="8.6"
2023-03-09T19:53:32.161 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el8"
2023-03-09T19:53:32.161 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-09T19:53:32.161 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31"
2023-03-09T19:53:32.161 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-09T19:53:32.161 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.redhat.com/"
2023-03-09T19:53:32.161 INFO:teuthology.orchestra.run.smithi156.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-09T19:53:32.161 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-09T19:53:32.162 INFO:teuthology.orchestra.run.smithi156.stdout:
2023-03-09T19:53:32.162 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-09T19:53:32.162 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-09T19:53:32.162 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-09T19:53:32.162 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-09T19:53:32.163 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server
2023-03-09T19:53:32.287 INFO:teuthology.dispatcher.supervisor:Running job 7198152
2023-03-09T19:53:32.288 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198152 --name pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} -- /home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198152/orig.config.yaml
2023-03-09T19:53:32.292 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198152
2023-03-09T19:53:32.293 INFO:teuthology.dispatcher.supervisor:Job PID: 13391
2023-03-09T19:53:32.293 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-10T07:53:55.761 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-10T07:53:56.063 INFO:teuthology.kill:Killing Pids: {13391}
2023-03-10T07:54:03.804 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-10T07:54:03.805 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-10T07:54:03.805 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-10T07:54:03.806 INFO:teuthology.misc:Compressing logs...
2023-03-10T07:54:03.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com'
2023-03-10T07:54:03.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com'
2023-03-10T07:54:04.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com'
2023-03-10T07:54:04.682 INFO:teuthology.misc:Compressing logs...
2023-03-10T07:54:05.061 INFO:teuthology.misc:Compressing logs...
2023-03-10T07:54:21.132 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2023-03-10T07:54:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.a.log: file size changed while zipping
2023-03-10T07:54:28.598 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2023-03-10T07:54:39.588 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2023-03-10T07:54:44.331 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2023-03-10T07:54:57.391 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.11.log: file size changed while zipping
2023-03-10T07:55:02.306 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2023-03-10T07:55:04.949 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mgr.x.log: file size changed while zipping
2023-03-10T07:55:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.10.log: file size changed while zipping
2023-03-10T07:55:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.h.log: file size changed while zipping
2023-03-10T07:55:22.511 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.8.log: file size changed while zipping
2023-03-10T07:55:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.e.log: file size changed while zipping
2023-03-10T07:55:26.339 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.f.log: file size changed while zipping
2023-03-10T07:55:29.132 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.9.log: file size changed while zipping
2023-03-10T07:55:29.886 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.c.log: file size changed while zipping
2023-03-10T07:55:34.319 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.j.log: file size changed while zipping
2023-03-10T07:55:47.482 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2023-03-10T07:56:19.442 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2023-03-10T07:57:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2023-03-10T07:57:10.138 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.k.log: file size changed while zipping
2023-03-10T07:57:26.470 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.i.log: file size changed while zipping
2023-03-10T07:57:31.876 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2023-03-10T07:57:32.536 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.l.log: file size changed while zipping
2023-03-10T07:58:03.590 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2023-03-10T07:58:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2023-03-10T07:58:48.543 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.b.log: file size changed while zipping
2023-03-10T08:00:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:tar: ./864788c8-beb5-11ed-9af2-001a4aab830c/ceph-osd.10.log.gz: file changed as we read it
2023-03-10T08:00:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:tar: ./864788c8-beb5-11ed-9af2-001a4aab830c: file changed as we read it
2023-03-10T08:01:29.300 INFO:teuthology.orchestra.run.smithi156.stderr:tar: ./864788c8-beb5-11ed-9af2-001a4aab830c/ceph-mds.i.log.gz: file changed as we read it
2023-03-10T08:02:29.376 INFO:teuthology.orchestra.run.smithi156.stderr:tar: ./864788c8-beb5-11ed-9af2-001a4aab830c: file changed as we read it
2023-03-10T08:02:29.501 INFO:teuthology.kill:No teuthology processes running
2023-03-10T08:02:30.298 INFO:teuthology.kill:Nuking machines: ['smithi007', 'smithi016', 'smithi156']
2023-03-10T08:02:30.907 INFO:teuthology.kill:2023-03-10 08:02:30,907.907 INFO:teuthology.nuke:targets:
2023-03-10T08:02:30.908 INFO:teuthology.kill:  smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-10T08:02:30.909 INFO:teuthology.kill:  smithi016.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-10T08:02:30.909 INFO:teuthology.kill:  smithi156.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-10T08:02:30.909 INFO:teuthology.kill:2023-03-10 08:02:30,907.907 INFO:teuthology.nuke:Checking targets against current locks
2023-03-10T08:02:30.975 INFO:teuthology.kill:2023-03-10 08:02:30,975.975 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-10T08:02:30.999 INFO:teuthology.kill:2023-03-10 08:02:30,999.999 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-10T08:02:31.023 INFO:teuthology.kill:2023-03-10 08:02:31,023.023 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-10T08:02:31.062 INFO:teuthology.kill:2023-03-10 08:02:31,062.062 INFO:teuthology.orchestra.console:Power off smithi007
2023-03-10T08:02:31.082 INFO:teuthology.kill:2023-03-10 08:02:31,081.081 INFO:teuthology.orchestra.console:Power off smithi016
2023-03-10T08:02:31.119 INFO:teuthology.kill:2023-03-10 08:02:31,119.119 INFO:teuthology.orchestra.console:Power off smithi156
2023-03-10T08:02:39.439 INFO:teuthology.kill:2023-03-10 08:02:39,439.439 INFO:teuthology.orchestra.console:Power off for smithi007 completed
2023-03-10T08:02:39.451 INFO:teuthology.kill:2023-03-10 08:02:39,450.450 INFO:teuthology.orchestra.console:Power off for smithi016 completed
2023-03-10T08:02:39.484 INFO:teuthology.kill:2023-03-10 08:02:39,484.484 INFO:teuthology.orchestra.console:Power off for smithi156 completed
2023-03-10T08:02:39.602 INFO:teuthology.kill:2023-03-10 08:02:39,601.601 INFO:teuthology.lock.ops:unlocked: smithi007.front.sepia.ceph.com
2023-03-10T08:02:39.611 INFO:teuthology.kill:2023-03-10 08:02:39,611.611 INFO:teuthology.lock.ops:unlocked: smithi016.front.sepia.ceph.com
2023-03-10T08:02:39.625 INFO:teuthology.kill:2023-03-10 08:02:39,624.624 INFO:teuthology.lock.ops:unlocked: smithi156.front.sepia.ceph.com
2023-03-10T08:04:39.874 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-10T08:04:40.014 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi007 but it was locked by another job: /home/teuthworker/archive/vshankar-2023-03-10_06:49:00-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7200631
2023-03-10T08:04:40.014 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi016 but it was locked by another job: /home/teuthworker/archive/vshankar-2023-03-10_06:49:00-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7200631
2023-03-10T08:04:40.014 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi156 but it was locked by another job: /home/teuthworker/archive/vshankar-2023-03-10_06:49:00-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7200632