2023-03-13T11:12:57.550 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f
2023-03-13T11:12:57.738 INFO:teuthology.lock.ops:Start node 'smithi082.front.sepia.ceph.com' reimaging
2023-03-13T11:12:57.739 INFO:teuthology.lock.ops:Updating [smithi082.front.sepia.ceph.com]: reset os type and version on server
2023-03-13T11:12:57.739 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server
2023-03-13T11:12:57.769 INFO:teuthology.lock.ops:Node 'smithi082.front.sepia.ceph.com' reimaging is complete
2023-03-13T11:12:57.770 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging
2023-03-13T11:12:57.770 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server
2023-03-13T11:12:57.770 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server
2023-03-13T11:12:57.794 INFO:teuthology.lock.ops:Node 'smithi150.front.sepia.ceph.com' reimaging is complete
2023-03-13T11:12:57.794 INFO:teuthology.lock.ops:Start node 'smithi181.front.sepia.ceph.com' reimaging
2023-03-13T11:12:57.794 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: reset os type and version on server
2023-03-13T11:12:57.794 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server
2023-03-13T11:12:57.819 INFO:teuthology.lock.ops:Node 'smithi181.front.sepia.ceph.com' reimaging is complete
2023-03-13T11:12:57.936 INFO:teuthology.provision.fog.smithi082:Scheduling deploy of ubuntu 20.04
2023-03-13T11:12:57.946 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of ubuntu 20.04
2023-03-13T11:12:57.988 INFO:teuthology.provision.fog.smithi181:Scheduling deploy of ubuntu 20.04
2023-03-13T11:12:58.171 INFO:teuthology.orchestra.console:Power off smithi150
2023-03-13T11:12:58.190 INFO:teuthology.orchestra.console:Power off smithi082
2023-03-13T11:12:58.232 INFO:teuthology.orchestra.console:Power off smithi181
2023-03-13T11:12:58.344 INFO:teuthology.orchestra.console:Power off for smithi082 completed
2023-03-13T11:12:58.445 INFO:teuthology.orchestra.console:Power on smithi082
2023-03-13T11:13:02.698 INFO:teuthology.orchestra.console:Power on for smithi082 completed
2023-03-13T11:13:02.799 INFO:teuthology.provision.fog.smithi082:Waiting for deploy to finish
2023-03-13T11:13:10.664 INFO:teuthology.orchestra.console:Power off for smithi150 completed
2023-03-13T11:13:10.709 INFO:teuthology.orchestra.console:Power off for smithi181 completed
2023-03-13T11:13:10.766 INFO:teuthology.orchestra.console:Power on smithi150
2023-03-13T11:13:10.811 INFO:teuthology.orchestra.console:Power on smithi181
2023-03-13T11:13:15.026 INFO:teuthology.orchestra.console:Power on for smithi150 completed
2023-03-13T11:13:15.127 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish
2023-03-13T11:13:19.175 INFO:teuthology.orchestra.console:Power on for smithi181 completed
2023-03-13T11:13:19.276 INFO:teuthology.provision.fog.smithi181:Waiting for deploy to finish
2023-03-13T11:16:51.913 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-03-13T11:17:01.001 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-03-13T11:17:04.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-03-13T11:17:08.329 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-13T11:17:10.053 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-03-13T11:17:13.385 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-03-13T11:17:17.381 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-13T11:17:19.109 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-03-13T11:17:22.437 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-03-13T11:17:26.441 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-13T11:17:28.165 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-03-13T11:17:31.493 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-03-13T11:17:35.493 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-13T11:17:40.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-03-13T11:17:44.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-13T11:17:49.605 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-03-13T11:17:53.605 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-13T11:17:58.661 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-03-13T11:18:34.186 WARNING:teuthology.provision.fog:timed out
2023-03-13T11:18:40.434 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-13T11:18:40.936 INFO:teuthology.provision.fog.smithi082:Node is ready
2023-03-13T11:18:40.942 INFO:teuthology.orchestra.run.smithi082.stdout:smithi082.front.sepia.ceph.com
2023-03-13T11:18:40.949 INFO:teuthology.orchestra.run.smithi082.stdout:172.21.15.82 smithi082.front.sepia.ceph.com smithi082
2023-03-13T11:18:41.142 INFO:teuthology.provision.fog.smithi082:Deploy complete!
2023-03-13T11:18:59.627 WARNING:teuthology.provision.fog:timed out
2023-03-13T11:19:04.671 WARNING:teuthology.provision.fog:timed out
2023-03-13T11:19:05.762 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-13T11:19:06.248 INFO:teuthology.provision.fog.smithi181:Node is ready
2023-03-13T11:19:06.255 INFO:teuthology.orchestra.run.smithi181.stdout:smithi181.front.sepia.ceph.com
2023-03-13T11:19:06.260 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.15.181 smithi181.front.sepia.ceph.com smithi181
2023-03-13T11:19:06.439 INFO:teuthology.provision.fog.smithi181:Deploy complete!
2023-03-13T11:19:10.885 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-13T11:19:11.442 INFO:teuthology.provision.fog.smithi150:Node is ready
2023-03-13T11:19:11.448 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150.front.sepia.ceph.com
2023-03-13T11:19:11.455 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150
2023-03-13T11:19:11.611 INFO:teuthology.provision.fog.smithi150:Deploy complete!
2023-03-13T11:19:11.975 INFO:teuthology.lock.ops:Checking smithi082.front.sepia.ceph.com
2023-03-13T11:19:11.976 INFO:teuthology.lock.ops:Checking smithi150.front.sepia.ceph.com
2023-03-13T11:19:11.976 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-13T11:19:12.007 INFO:teuthology.lock.ops:Checking smithi181.front.sepia.ceph.com
2023-03-13T11:19:12.009 INFO:teuthology.lock.ops:Updating [smithi082.front.sepia.ceph.com]: set os type and version on server
2023-03-13T11:19:12.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-13T11:19:12.264 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64
2023-03-13T11:19:12.270 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="Ubuntu"
2023-03-13T11:19:12.270 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-03-13T11:19:12.270 INFO:teuthology.orchestra.run.smithi082.stdout:ID=ubuntu
2023-03-13T11:19:12.270 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE=debian
2023-03-13T11:19:12.271 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-03-13T11:19:12.271 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="20.04"
2023-03-13T11:19:12.271 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://www.ubuntu.com/"
2023-03-13T11:19:12.271 INFO:teuthology.orchestra.run.smithi082.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-03-13T11:19:12.271 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-03-13T11:19:12.272 INFO:teuthology.orchestra.run.smithi082.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-03-13T11:19:12.272 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_CODENAME=focal
2023-03-13T11:19:12.272 INFO:teuthology.orchestra.run.smithi082.stdout:UBUNTU_CODENAME=focal
2023-03-13T11:19:12.273 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server
2023-03-13T11:19:12.298 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: set os type and version on server
2023-03-13T11:19:12.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com'
2023-03-13T11:19:12.480 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64
2023-03-13T11:19:12.486 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Ubuntu"
2023-03-13T11:19:12.487 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-03-13T11:19:12.487 INFO:teuthology.orchestra.run.smithi150.stdout:ID=ubuntu
2023-03-13T11:19:12.487 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE=debian
2023-03-13T11:19:12.488 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-03-13T11:19:12.488 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="20.04"
2023-03-13T11:19:12.488 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.ubuntu.com/"
2023-03-13T11:19:12.488 INFO:teuthology.orchestra.run.smithi150.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-03-13T11:19:12.489 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-03-13T11:19:12.489 INFO:teuthology.orchestra.run.smithi150.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-03-13T11:19:12.489 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_CODENAME=focal
2023-03-13T11:19:12.490 INFO:teuthology.orchestra.run.smithi150.stdout:UBUNTU_CODENAME=focal
2023-03-13T11:19:12.490 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server
2023-03-13T11:19:12.515 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: set os type and version on server
2023-03-13T11:19:12.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-13T11:19:12.692 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64
2023-03-13T11:19:12.697 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Ubuntu"
2023-03-13T11:19:12.697 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-03-13T11:19:12.698 INFO:teuthology.orchestra.run.smithi181.stdout:ID=ubuntu
2023-03-13T11:19:12.698 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE=debian
2023-03-13T11:19:12.698 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-03-13T11:19:12.698 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="20.04"
2023-03-13T11:19:12.698 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.ubuntu.com/"
2023-03-13T11:19:12.698 INFO:teuthology.orchestra.run.smithi181.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-03-13T11:19:12.699 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-03-13T11:19:12.699 INFO:teuthology.orchestra.run.smithi181.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-03-13T11:19:12.699 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_CODENAME=focal
2023-03-13T11:19:12.699 INFO:teuthology.orchestra.run.smithi181.stdout:UBUNTU_CODENAME=focal
2023-03-13T11:19:12.699 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server
2023-03-13T11:19:12.823 INFO:teuthology.dispatcher.supervisor:Running job 7204975
2023-03-13T11:19:12.823 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2023-03-13_02:28:24-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7204975 --name vshankar-2023-03-13_02:28:24-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi --description fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} -- /home/teuthworker/archive/vshankar-2023-03-13_02:28:24-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7204975/orig.config.yaml
2023-03-13T11:19:12.827 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2023-03-13_02:28:24-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7204975
2023-03-13T11:19:12.828 INFO:teuthology.dispatcher.supervisor:Job PID: 21500
2023-03-13T11:19:12.828 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-13T23:20:10.681 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-13T23:20:10.846 INFO:teuthology.kill:Killing Pids: {21500}
2023-03-13T23:20:10.876 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
KeyError: 'status'
2023-03-13T23:20:10.907 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
2023-03-13T23:20:10.907 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-03-13T23:20:10.908 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['client.0']
2023-03-13T23:20:10.908 INFO:teuthology.misc:Compressing logs...
2023-03-13T23:20:10.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-13T23:20:11.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com'
2023-03-13T23:20:11.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-13T23:20:11.836 INFO:teuthology.misc:Compressing logs...
2023-03-13T23:20:12.055 INFO:teuthology.misc:Compressing logs...
2023-03-13T23:23:43.735 INFO:teuthology.kill:No teuthology processes running
2023-03-13T23:23:43.736 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-13T23:25:43.961 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-13T23:25:44.092 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-13T23:25:44.093 INFO:teuthology.nuke:Checking targets against current locks
2023-03-13T23:25:44.152 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-13T23:25:44.175 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-13T23:25:44.192 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-13T23:25:44.231 INFO:teuthology.orchestra.console:Power off smithi082
2023-03-13T23:25:44.247 INFO:teuthology.orchestra.console:Power off smithi150
2023-03-13T23:25:44.287 INFO:teuthology.orchestra.console:Power off smithi181
2023-03-13T23:25:52.609 INFO:teuthology.orchestra.console:Power off for smithi082 completed
2023-03-13T23:25:52.657 INFO:teuthology.orchestra.console:Power off for smithi181 completed
2023-03-13T23:25:52.767 INFO:teuthology.lock.ops:unlocked: smithi082.front.sepia.ceph.com
2023-03-13T23:25:52.809 INFO:teuthology.lock.ops:unlocked: smithi181.front.sepia.ceph.com
2023-03-13T23:25:56.732 INFO:teuthology.orchestra.console:Power off for smithi150 completed
2023-03-13T23:25:56.900 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com