2022-05-02T17:17:30.284 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166
2022-05-02T17:17:30.440 INFO:teuthology.lock.ops:Start node 'smithi057.front.sepia.ceph.com' reimaging
2022-05-02T17:17:30.440 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: reset os type and version on server
2022-05-02T17:17:30.441 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2022-05-02T17:17:30.462 INFO:teuthology.lock.ops:Node 'smithi057.front.sepia.ceph.com' reimaging is complete
2022-05-02T17:17:30.462 INFO:teuthology.lock.ops:Start node 'smithi097.front.sepia.ceph.com' reimaging
2022-05-02T17:17:30.462 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: reset os type and version on server
2022-05-02T17:17:30.462 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server
2022-05-02T17:17:30.480 INFO:teuthology.lock.ops:Node 'smithi097.front.sepia.ceph.com' reimaging is complete
2022-05-02T17:17:30.480 INFO:teuthology.lock.ops:Start node 'smithi170.front.sepia.ceph.com' reimaging
2022-05-02T17:17:30.480 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: reset os type and version on server
2022-05-02T17:17:30.480 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2022-05-02T17:17:30.498 INFO:teuthology.lock.ops:Node 'smithi170.front.sepia.ceph.com' reimaging is complete
2022-05-02T17:17:30.685 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of centos 8.stream
2022-05-02T17:17:30.709 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of centos 8.stream
2022-05-02T17:17:30.711 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of centos 8.stream
2022-05-02T17:17:30.997 INFO:teuthology.orchestra.console:Power off smithi057
2022-05-02T17:17:31.013 INFO:teuthology.orchestra.console:Power off smithi097
2022-05-02T17:17:31.059 INFO:teuthology.orchestra.console:Power off smithi170
2022-05-02T17:17:39.418 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2022-05-02T17:17:39.519 INFO:teuthology.orchestra.console:Power on smithi170
2022-05-02T17:17:43.480 INFO:teuthology.orchestra.console:Power off for smithi057 completed
2022-05-02T17:17:43.493 INFO:teuthology.orchestra.console:Power off for smithi097 completed
2022-05-02T17:17:43.581 INFO:teuthology.orchestra.console:Power on smithi057
2022-05-02T17:17:43.593 INFO:teuthology.orchestra.console:Power on smithi097
2022-05-02T17:17:47.886 INFO:teuthology.orchestra.console:Power on for smithi170 completed
2022-05-02T17:17:47.987 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish
2022-05-02T17:17:51.949 INFO:teuthology.orchestra.console:Power on for smithi057 completed
2022-05-02T17:17:51.997 INFO:teuthology.orchestra.console:Power on for smithi097 completed
2022-05-02T17:17:52.075 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish
2022-05-02T17:17:52.098 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish
2022-05-02T17:22:24.065 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-05-02T17:22:24.367 INFO:teuthology.provision.fog.smithi097:Node is ready
2022-05-02T17:22:24.424 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097.front.sepia.ceph.com
2022-05-02T17:22:24.481 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97 smithi097.front.sepia.ceph.com smithi097
2022-05-02T17:22:24.825 INFO:teuthology.provision.fog.smithi097:Deploy complete!
2022-05-02T17:22:26.172 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-05-02T17:22:26.419 INFO:teuthology.provision.fog.smithi170:Node is ready
2022-05-02T17:22:26.477 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com
2022-05-02T17:22:26.536 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170
2022-05-02T17:22:26.886 INFO:teuthology.provision.fog.smithi170:Deploy complete!
2022-05-02T17:22:30.251 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-05-02T17:22:30.505 INFO:teuthology.provision.fog.smithi057:Node is ready
2022-05-02T17:22:30.523 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057.front.sepia.ceph.com
2022-05-02T17:22:30.582 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57 smithi057.front.sepia.ceph.com smithi057
2022-05-02T17:22:30.947 INFO:teuthology.provision.fog.smithi057:Deploy complete!
2022-05-02T17:22:31.129 INFO:teuthology.lock.ops:Checking smithi057.front.sepia.ceph.com
2022-05-02T17:22:31.129 INFO:teuthology.lock.ops:New key found. Updating...
2022-05-02T17:22:31.148 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com
2022-05-02T17:22:31.148 INFO:teuthology.lock.ops:New key found. Updating...
2022-05-02T17:22:31.166 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com
2022-05-02T17:22:31.166 INFO:teuthology.lock.ops:New key found. Updating...
2022-05-02T17:22:31.184 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: set os type and version on server
2022-05-02T17:22:31.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2022-05-02T17:22:31.385 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64
2022-05-02T17:22:31.442 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream"
2022-05-02T17:22:31.442 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="8"
2022-05-02T17:22:31.442 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="8"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el8"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 8"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-05-02T17:22:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-05-02T17:22:31.445 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2022-05-02T17:22:31.462 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server
2022-05-02T17:22:31.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com'
2022-05-02T17:22:31.667 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64
2022-05-02T17:22:31.727 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream"
2022-05-02T17:22:31.727 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8"
2022-05-02T17:22:31.727 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos"
2022-05-02T17:22:31.727 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora"
2022-05-02T17:22:31.727 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8"
2022-05-02T17:22:31.727 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8"
2022-05-02T17:22:31.727 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 8"
2022-05-02T17:22:31.728 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31"
2022-05-02T17:22:31.728 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-05-02T17:22:31.728 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/"
2022-05-02T17:22:31.728 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-05-02T17:22:31.728 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-05-02T17:22:31.728 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-05-02T17:22:31.729 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server
2022-05-02T17:22:31.747 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server
2022-05-02T17:22:31.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2022-05-02T17:22:31.960 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64
2022-05-02T17:22:32.019 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream"
2022-05-02T17:22:32.019 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8"
2022-05-02T17:22:32.019 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos"
2022-05-02T17:22:32.019 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora"
2022-05-02T17:22:32.020 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8"
2022-05-02T17:22:32.020 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8"
2022-05-02T17:22:32.020 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 8"
2022-05-02T17:22:32.020 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31"
2022-05-02T17:22:32.020 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-05-02T17:22:32.020 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/"
2022-05-02T17:22:32.020 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-05-02T17:22:32.020 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-05-02T17:22:32.020 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-05-02T17:22:32.021 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2022-05-02T17:22:32.122 INFO:teuthology.dispatcher.supervisor:Running job 6818714
2022-05-02T17:22:32.122 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-05-02_16:58:59-fs-wip-vshankar-testing1-20220502-201957-testing-default-smithi/6818714 --name vshankar-2022-05-02_16:58:59-fs-wip-vshankar-testing1-20220502-201957-testing-default-smithi --description fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} -- /home/teuthworker/archive/vshankar-2022-05-02_16:58:59-fs-wip-vshankar-testing1-20220502-201957-testing-default-smithi/6818714/orig.config.yaml
2022-05-02T17:22:32.126 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-05-02_16:58:59-fs-wip-vshankar-testing1-20220502-201957-testing-default-smithi/6818714
2022-05-02T17:22:32.127 INFO:teuthology.dispatcher.supervisor:Job PID: 31351
2022-05-02T17:22:32.127 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-05-02T23:52:59.374 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-05-02T23:52:59.557 INFO:teuthology.kill:Killing Pids: {31351}
2022-05-02T23:52:59.571 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/dispatcher/supervisor.py", line 282, in run_with_watchdog
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/kill.py", line 90, in kill_job
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/kill.py", line 227, in find_targets
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'teuthology-lock': 'teuthology-lock'
2022-05-02T23:52:59.612 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
2022-05-02T23:52:59.612 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['client.0']
2022-05-02T23:52:59.613 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['client.1']
2022-05-02T23:52:59.613 INFO:teuthology.misc:Compressing logs...
2022-05-02T23:52:59.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2022-05-02T23:52:59.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com'
2022-05-02T23:53:00.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2022-05-02T23:53:00.862 INFO:teuthology.misc:Compressing logs...
2022-05-02T23:53:21.608 INFO:teuthology.misc:Compressing logs...
2022-05-02T23:54:43.290 INFO:teuthology.kill:No teuthology processes running
2022-05-02T23:54:43.295 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/kill.py", line 90, in kill_job
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/teuthology/kill.py", line 227, in find_targets
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'teuthology-lock': 'teuthology-lock'
2022-05-02T23:56:43.518 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15