2023-03-12T13:58:44.229 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f
2023-03-12T13:58:44.365 INFO:teuthology.lock.ops:Start node 'smithi007.front.sepia.ceph.com' reimaging
2023-03-12T13:58:44.366 INFO:teuthology.lock.ops:Updating [smithi007.front.sepia.ceph.com]: reset os type and version on server
2023-03-12T13:58:44.366 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server
2023-03-12T13:58:44.390 INFO:teuthology.lock.ops:Node 'smithi007.front.sepia.ceph.com' reimaging is complete
2023-03-12T13:58:44.390 INFO:teuthology.lock.ops:Start node 'smithi043.front.sepia.ceph.com' reimaging
2023-03-12T13:58:44.390 INFO:teuthology.lock.ops:Updating [smithi043.front.sepia.ceph.com]: reset os type and version on server
2023-03-12T13:58:44.390 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server
2023-03-12T13:58:44.414 INFO:teuthology.lock.ops:Node 'smithi043.front.sepia.ceph.com' reimaging is complete
2023-03-12T13:58:44.534 INFO:teuthology.provision.fog.smithi007:Scheduling deploy of centos 8.stream
2023-03-12T13:58:44.565 INFO:teuthology.provision.fog.smithi043:Scheduling deploy of centos 8.stream
2023-03-12T13:58:44.767 INFO:teuthology.orchestra.console:Power off smithi043
2023-03-12T13:58:44.785 INFO:teuthology.orchestra.console:Power off smithi007
2023-03-12T13:58:44.916 INFO:teuthology.orchestra.console:Power off for smithi043 completed
2023-03-12T13:58:44.929 INFO:teuthology.orchestra.console:Power off for smithi007 completed
2023-03-12T13:58:45.018 INFO:teuthology.orchestra.console:Power on smithi043
2023-03-12T13:58:45.031 INFO:teuthology.orchestra.console:Power on smithi007
2023-03-12T13:58:49.287 INFO:teuthology.orchestra.console:Power on for smithi007 completed
2023-03-12T13:58:49.388 INFO:teuthology.provision.fog.smithi007:Waiting for deploy to finish
2023-03-12T13:58:53.393 INFO:teuthology.orchestra.console:Power on for smithi043 completed
2023-03-12T13:58:53.495 INFO:teuthology.provision.fog.smithi043:Waiting for deploy to finish
2023-03-12T14:01:24.414 ERROR:teuthology.orchestra.connection:Error authenticating with smithi043.front.sepia.ceph.com: Authentication failed.
2023-03-12T14:01:53.925 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.7
2023-03-12T14:02:02.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.7
2023-03-12T14:02:12.037 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.7
2023-03-12T14:02:25.418 WARNING:teuthology.provision.fog:timed out
2023-03-12T14:02:49.989 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.43
2023-03-12T14:02:59.045 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.43
2023-03-12T14:03:05.392 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-12T14:03:18.040 WARNING:teuthology.provision.fog:timed out
2023-03-12T14:03:20.204 INFO:teuthology.provision.fog.smithi043:Node is ready
2023-03-12T14:03:20.228 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043.front.sepia.ceph.com
2023-03-12T14:03:20.287 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43 smithi043.front.sepia.ceph.com smithi043
2023-03-12T14:03:20.689 INFO:teuthology.provision.fog.smithi043:Deploy complete!
2023-03-12T14:03:24.373 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-12T14:03:24.624 INFO:teuthology.provision.fog.smithi007:Node is ready
2023-03-12T14:03:24.683 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007.front.sepia.ceph.com
2023-03-12T14:03:24.743 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7 smithi007.front.sepia.ceph.com smithi007
2023-03-12T14:03:25.115 INFO:teuthology.provision.fog.smithi007:Deploy complete!
2023-03-12T14:03:25.304 INFO:teuthology.lock.ops:Checking smithi007.front.sepia.ceph.com
2023-03-12T14:03:25.305 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-12T14:03:25.612 INFO:teuthology.lock.ops:Checking smithi043.front.sepia.ceph.com
2023-03-12T14:03:25.612 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-12T14:03:25.637 INFO:teuthology.lock.ops:Updating [smithi007.front.sepia.ceph.com]: set os type and version on server
2023-03-12T14:03:25.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com'
2023-03-12T14:03:25.829 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64
2023-03-12T14:03:25.887 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream"
2023-03-12T14:03:25.887 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="8"
2023-03-12T14:03:25.887 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos"
2023-03-12T14:03:25.888 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora"
2023-03-12T14:03:25.888 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="8"
2023-03-12T14:03:25.888 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el8"
2023-03-12T14:03:25.888 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-12T14:03:25.888 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31"
2023-03-12T14:03:25.888 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-12T14:03:25.888 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/"
2023-03-12T14:03:25.889 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-12T14:03:25.889 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-12T14:03:25.889 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-12T14:03:25.890 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server
2023-03-12T14:03:25.920 INFO:teuthology.lock.ops:Updating [smithi043.front.sepia.ceph.com]: set os type and version on server
2023-03-12T14:03:25.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com'
2023-03-12T14:03:26.118 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64
2023-03-12T14:03:26.179 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream"
2023-03-12T14:03:26.179 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="8"
2023-03-12T14:03:26.180 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos"
2023-03-12T14:03:26.180 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora"
2023-03-12T14:03:26.180 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="8"
2023-03-12T14:03:26.180 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el8"
2023-03-12T14:03:26.180 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-12T14:03:26.180 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31"
2023-03-12T14:03:26.180 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-12T14:03:26.181 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/"
2023-03-12T14:03:26.181 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-12T14:03:26.181 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-12T14:03:26.181 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-12T14:03:26.181 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server
2023-03-12T14:03:26.292 INFO:teuthology.dispatcher.supervisor:Running job 7204120
2023-03-12T14:03:26.292 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_rzarzynski@teuthology --archive /home/teuthworker/archive/rzarzynski-2023-03-12_13:57:50-crimson-rados-main-distro-crimson-smithi/7204120 --name rzarzynski-2023-03-12_13:57:50-crimson-rados-main-distro-crimson-smithi --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} -- /home/teuthworker/archive/rzarzynski-2023-03-12_13:57:50-crimson-rados-main-distro-crimson-smithi/7204120/orig.config.yaml
2023-03-12T14:03:26.295 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rzarzynski-2023-03-12_13:57:50-crimson-rados-main-distro-crimson-smithi/7204120
2023-03-12T14:03:26.296 INFO:teuthology.dispatcher.supervisor:Job PID: 12605
2023-03-12T14:03:26.297 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-13T02:03:49.824 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-13T02:03:49.950 INFO:teuthology.kill:Killing Pids: {12605}
2023-03-13T02:03:49.980 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
KeyError: 'status'
2023-03-13T02:03:50.006 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a']
2023-03-13T02:03:50.007 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']
2023-03-13T02:03:50.007 INFO:teuthology.misc:Compressing logs...
2023-03-13T02:03:50.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com'
2023-03-13T02:03:50.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com'
2023-03-13T02:03:50.534 INFO:teuthology.misc:Compressing logs...
2023-03-13T02:03:50.692 INFO:teuthology.misc:Compressing logs...
2023-03-13T02:03:50.887 INFO:teuthology.kill:No teuthology processes running
2023-03-13T02:03:50.888 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-13T02:05:51.019 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-13T02:05:51.120 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-13T02:05:51.120 INFO:teuthology.nuke:Checking targets against current locks
2023-03-13T02:05:51.167 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-13T02:05:51.194 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-13T02:05:51.246 INFO:teuthology.orchestra.console:Power off smithi007
2023-03-13T02:05:51.286 INFO:teuthology.orchestra.console:Power off smithi043
2023-03-13T02:05:59.622 INFO:teuthology.orchestra.console:Power off for smithi007 completed
2023-03-13T02:05:59.653 INFO:teuthology.orchestra.console:Power off for smithi043 completed
2023-03-13T02:05:59.791 INFO:teuthology.lock.ops:unlocked: smithi007.front.sepia.ceph.com
2023-03-13T02:05:59.807 INFO:teuthology.lock.ops:unlocked: smithi043.front.sepia.ceph.com