2023-03-19T23:59:54.406 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3
2023-03-19T23:59:56.773 INFO:teuthology.lock.ops:Start node 'smithi062.front.sepia.ceph.com' reimaging
2023-03-19T23:59:58.280 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: reset os type and version on server
2023-03-19T23:59:58.280 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server
2023-03-19T23:59:58.301 INFO:teuthology.lock.ops:Node 'smithi062.front.sepia.ceph.com' reimaging is complete
2023-03-19T23:59:59.510 INFO:teuthology.lock.ops:Start node 'smithi118.front.sepia.ceph.com' reimaging
2023-03-19T23:59:59.510 INFO:teuthology.lock.ops:Updating [smithi118.front.sepia.ceph.com]: reset os type and version on server
2023-03-19T23:59:59.510 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server
2023-03-19T23:59:59.533 INFO:teuthology.lock.ops:Node 'smithi118.front.sepia.ceph.com' reimaging is complete
2023-03-19T23:59:59.878 INFO:teuthology.lock.ops:Start node 'smithi170.front.sepia.ceph.com' reimaging
2023-03-20T00:00:00.769 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: reset os type and version on server
2023-03-20T00:00:00.773 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2023-03-20T00:00:00.800 INFO:teuthology.lock.ops:Node 'smithi170.front.sepia.ceph.com' reimaging is complete
2023-03-20T00:00:01.753 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of rhel 8.6
2023-03-20T00:00:03.385 INFO:teuthology.provision.fog.smithi118:Scheduling deploy of rhel 8.6
2023-03-20T00:00:04.640 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of rhel 8.6
2023-03-20T00:00:05.805 INFO:teuthology.orchestra.console:Power off smithi062
2023-03-20T00:00:06.199 INFO:teuthology.orchestra.console:Power off smithi118
2023-03-20T00:00:09.103 INFO:teuthology.orchestra.console:Power off smithi170
2023-03-20T00:00:14.888 INFO:teuthology.orchestra.console:Power off for smithi062 completed
2023-03-20T00:00:17.126 INFO:teuthology.orchestra.console:Power on smithi062
2023-03-20T00:00:20.225 INFO:teuthology.orchestra.console:Power off for smithi118 completed
2023-03-20T00:00:21.404 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2023-03-20T00:00:21.504 INFO:teuthology.orchestra.console:Power on smithi118
2023-03-20T00:00:22.881 INFO:teuthology.orchestra.console:Power on smithi170
2023-03-20T00:00:25.046 INFO:teuthology.orchestra.console:Power on for smithi062 completed
2023-03-20T00:00:25.804 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish
2023-03-20T00:00:29.339 INFO:teuthology.orchestra.console:Power on for smithi118 completed
2023-03-20T00:00:30.232 INFO:teuthology.orchestra.console:Power on for smithi170 completed
2023-03-20T00:00:33.578 INFO:teuthology.provision.fog.smithi118:Waiting for deploy to finish
2023-03-20T00:00:33.678 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish
2023-03-20T00:03:24.037 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-03-20T00:03:33.253 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.62
2023-03-20T00:03:41.194 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-03-20T00:04:17.541 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-20T00:04:26.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-20T00:04:32.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.62
2023-03-20T00:04:38.129 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-20T00:04:39.685 INFO:teuthology.provision.fog.smithi170:Node is ready
2023-03-20T00:04:41.219 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-20T00:04:42.736 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com
2023-03-20T00:04:43.826 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170
2023-03-20T00:04:45.093 INFO:teuthology.provision.fog.smithi170:Deploy complete!
2023-03-20T00:04:45.946 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-20T00:04:47.018 INFO:teuthology.provision.fog.smithi062:Node is ready
2023-03-20T00:04:47.934 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com
2023-03-20T00:04:49.896 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062
2023-03-20T00:04:51.234 INFO:teuthology.provision.fog.smithi062:Deploy complete!
2023-03-20T00:05:08.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-20T00:05:19.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-20T00:05:26.495 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-20T00:05:28.263 INFO:teuthology.provision.fog.smithi118:Node is ready
2023-03-20T00:05:30.033 INFO:teuthology.orchestra.run.smithi118.stdout:smithi118.front.sepia.ceph.com
2023-03-20T00:05:31.826 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118 smithi118.front.sepia.ceph.com smithi118
2023-03-20T00:05:33.280 INFO:teuthology.provision.fog.smithi118:Deploy complete!
2023-03-20T00:05:33.951 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com
2023-03-20T00:05:34.421 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-20T00:05:34.448 INFO:teuthology.lock.ops:Checking smithi118.front.sepia.ceph.com
2023-03-20T00:05:35.353 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-20T00:05:35.381 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com
2023-03-20T00:05:36.243 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-20T00:05:36.271 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server
2023-03-20T00:05:36.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com'
2023-03-20T00:05:37.122 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64
2023-03-20T00:05:37.666 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Red Hat Enterprise Linux"
2023-03-20T00:05:38.182 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8.6 (Ootpa)"
2023-03-20T00:05:38.182 INFO:teuthology.orchestra.run.smithi062.stdout:ID="rhel"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="fedora"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8.6"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.redhat.com/"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-20T00:05:38.183 INFO:teuthology.orchestra.run.smithi062.stdout:
2023-03-20T00:05:38.615 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-20T00:05:38.615 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-20T00:05:38.615 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-20T00:05:38.615 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-20T00:05:38.616 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server
2023-03-20T00:05:38.639 INFO:teuthology.lock.ops:Updating [smithi118.front.sepia.ceph.com]: set os type and version on server
2023-03-20T00:05:39.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-20T00:05:39.221 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64
2023-03-20T00:05:40.270 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="Red Hat Enterprise Linux"
2023-03-20T00:05:41.564 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8.6 (Ootpa)"
2023-03-20T00:05:41.564 INFO:teuthology.orchestra.run.smithi118.stdout:ID="rhel"
2023-03-20T00:05:41.564 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="fedora"
2023-03-20T00:05:41.564 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8.6"
2023-03-20T00:05:41.564 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8"
2023-03-20T00:05:41.564 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-20T00:05:41.564 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31"
2023-03-20T00:05:41.565 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-20T00:05:41.565 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://www.redhat.com/"
2023-03-20T00:05:41.565 INFO:teuthology.orchestra.run.smithi118.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-20T00:05:41.565 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-20T00:05:41.565 INFO:teuthology.orchestra.run.smithi118.stdout:
2023-03-20T00:05:41.565 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-20T00:05:41.565 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-20T00:05:41.565 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-20T00:05:41.565 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-20T00:05:42.181 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server
2023-03-20T00:05:43.805 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server
2023-03-20T00:05:44.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-20T00:05:45.215 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64
2023-03-20T00:05:46.402 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="Red Hat Enterprise Linux"
2023-03-20T00:05:48.310 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8.6 (Ootpa)"
2023-03-20T00:05:48.310 INFO:teuthology.orchestra.run.smithi170.stdout:ID="rhel"
2023-03-20T00:05:48.310 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="fedora"
2023-03-20T00:05:48.310 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8.6"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://www.redhat.com/"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-20T00:05:48.311 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-20T00:05:48.312 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2023-03-20T00:05:48.435 INFO:teuthology.dispatcher.supervisor:Running job 7212528
2023-03-20T00:05:52.132 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-17_23:43:03-fs-reef-distro-default-smithi/7212528 --name yuriw-2023-03-17_23:43:03-fs-reef-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} -- /home/teuthworker/archive/yuriw-2023-03-17_23:43:03-fs-reef-distro-default-smithi/7212528/orig.config.yaml
2023-03-20T00:05:52.135 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-17_23:43:03-fs-reef-distro-default-smithi/7212528
2023-03-20T00:05:55.008 INFO:teuthology.dispatcher.supervisor:Job PID: 11416
2023-03-20T00:05:55.008 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-20T12:06:46.971 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-20T12:06:48.299 INFO:teuthology.kill:Killing Pids: {11416}
2023-03-20T12:06:50.091 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-20T12:06:50.837 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-20T12:06:50.838 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-20T12:06:50.838 INFO:teuthology.misc:Compressing logs...
2023-03-20T12:06:50.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com'
2023-03-20T12:06:51.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-20T12:06:51.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-20T12:06:53.693 INFO:teuthology.misc:Compressing logs...
2023-03-20T12:06:55.494 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-20T12:06:56.135 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-20T12:06:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-20T12:06:57.451 INFO:teuthology.misc:Compressing logs...
2023-03-20T12:08:22.306 INFO:teuthology.kill:No teuthology processes running
2023-03-20T12:08:23.041 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-20T12:10:24.028 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-20T12:10:27.596 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-20T12:10:29.224 INFO:teuthology.nuke:Checking targets against current locks
2023-03-20T12:10:29.287 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-20T12:10:30.410 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-20T12:10:31.665 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-20T12:10:32.726 INFO:teuthology.orchestra.console:Power off smithi062
2023-03-20T12:10:33.834 INFO:teuthology.orchestra.console:Power off smithi170
2023-03-20T12:10:35.001 INFO:teuthology.orchestra.console:Power off smithi118
2023-03-20T12:10:44.415 INFO:teuthology.orchestra.console:Power off for smithi062 completed
2023-03-20T12:10:44.818 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2023-03-20T12:10:44.818 INFO:teuthology.orchestra.console:Power off for smithi118 completed
2023-03-20T12:10:44.974 INFO:teuthology.lock.ops:unlocked: smithi062.front.sepia.ceph.com
2023-03-20T12:10:45.436 INFO:teuthology.lock.ops:unlocked: smithi118.front.sepia.ceph.com
2023-03-20T12:10:46.007 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com