2022-07-03T15:03:31.045 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3
2022-07-03T15:03:31.195 INFO:teuthology.lock.ops:Start node 'smithi157.front.sepia.ceph.com' reimaging
2022-07-03T15:03:31.197 INFO:teuthology.lock.ops:Updating [smithi157.front.sepia.ceph.com]: reset os type and version on server
2022-07-03T15:03:31.197 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server
2022-07-03T15:03:31.220 INFO:teuthology.lock.ops:Node 'smithi157.front.sepia.ceph.com' reimaging is complete
2022-07-03T15:03:31.220 INFO:teuthology.lock.ops:Start node 'smithi195.front.sepia.ceph.com' reimaging
2022-07-03T15:03:31.221 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: reset os type and version on server
2022-07-03T15:03:31.221 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server
2022-07-03T15:03:31.245 INFO:teuthology.lock.ops:Node 'smithi195.front.sepia.ceph.com' reimaging is complete
2022-07-03T15:03:31.446 INFO:teuthology.provision.fog.smithi157:Scheduling deploy of rhel 8.6
2022-07-03T15:03:31.449 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of rhel 8.6
2022-07-03T15:03:31.772 INFO:teuthology.orchestra.console:Power off smithi195
2022-07-03T15:03:31.817 INFO:teuthology.orchestra.console:Power off smithi157
2022-07-03T15:03:40.151 INFO:teuthology.orchestra.console:Power off for smithi195 completed
2022-07-03T15:03:40.184 INFO:teuthology.orchestra.console:Power off for smithi157 completed
2022-07-03T15:03:40.253 INFO:teuthology.orchestra.console:Power on smithi195
2022-07-03T15:03:40.286 INFO:teuthology.orchestra.console:Power on smithi157
2022-07-03T15:03:48.657 INFO:teuthology.orchestra.console:Power on for smithi195 completed
2022-07-03T15:03:48.662 INFO:teuthology.orchestra.console:Power on for smithi157 completed
2022-07-03T15:03:48.759 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish
2022-07-03T15:03:48.763 INFO:teuthology.provision.fog.smithi157:Waiting for deploy to finish
2022-07-03T15:08:27.023 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-07-03T15:08:27.284 INFO:teuthology.provision.fog.smithi195:Node is ready
2022-07-03T15:08:27.302 INFO:teuthology.orchestra.run.smithi195.stdout:smithi195.front.sepia.ceph.com
2022-07-03T15:08:27.360 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.15.195 smithi195.front.sepia.ceph.com smithi195
2022-07-03T15:08:27.694 INFO:teuthology.provision.fog.smithi195:Deploy complete!
2022-07-03T15:09:15.335 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-07-03T15:09:15.590 INFO:teuthology.provision.fog.smithi157:Node is ready
2022-07-03T15:09:15.608 INFO:teuthology.orchestra.run.smithi157.stdout:smithi157.front.sepia.ceph.com
2022-07-03T15:09:15.666 INFO:teuthology.orchestra.run.smithi157.stdout:172.21.15.157 smithi157.front.sepia.ceph.com smithi157
2022-07-03T15:09:16.010 INFO:teuthology.provision.fog.smithi157:Deploy complete!
2022-07-03T15:09:16.281 INFO:teuthology.lock.ops:Checking smithi157.front.sepia.ceph.com
2022-07-03T15:09:16.282 INFO:teuthology.lock.ops:New key found. Updating...
2022-07-03T15:09:16.309 INFO:teuthology.lock.ops:Checking smithi195.front.sepia.ceph.com
2022-07-03T15:09:16.309 INFO:teuthology.lock.ops:New key found. Updating...
2022-07-03T15:09:16.336 INFO:teuthology.lock.ops:Updating [smithi157.front.sepia.ceph.com]: set os type and version on server
2022-07-03T15:09:16.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com'
2022-07-03T15:09:16.544 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64
2022-07-03T15:09:16.601 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="Red Hat Enterprise Linux"
2022-07-03T15:09:16.602 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="8.6 (Ootpa)"
2022-07-03T15:09:16.602 INFO:teuthology.orchestra.run.smithi157.stdout:ID="rhel"
2022-07-03T15:09:16.602 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="fedora"
2022-07-03T15:09:16.602 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="8.6"
2022-07-03T15:09:16.603 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el8"
2022-07-03T15:09:16.603 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-07-03T15:09:16.603 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31"
2022-07-03T15:09:16.603 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-07-03T15:09:16.603 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://www.redhat.com/"
2022-07-03T15:09:16.604 INFO:teuthology.orchestra.run.smithi157.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-07-03T15:09:16.604 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-07-03T15:09:16.604 INFO:teuthology.orchestra.run.smithi157.stdout:
2022-07-03T15:09:16.604 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-07-03T15:09:16.605 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-07-03T15:09:16.605 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-07-03T15:09:16.605 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2022-07-03T15:09:16.607 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server
2022-07-03T15:09:16.626 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: set os type and version on server
2022-07-03T15:09:16.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com'
2022-07-03T15:09:16.902 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64
2022-07-03T15:09:16.960 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="Red Hat Enterprise Linux"
2022-07-03T15:09:16.961 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8.6 (Ootpa)"
2022-07-03T15:09:16.961 INFO:teuthology.orchestra.run.smithi195.stdout:ID="rhel"
2022-07-03T15:09:16.961 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="fedora"
2022-07-03T15:09:16.962 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8.6"
2022-07-03T15:09:16.962 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8"
2022-07-03T15:09:16.962 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-07-03T15:09:16.963 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31"
2022-07-03T15:09:16.963 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-07-03T15:09:16.963 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://www.redhat.com/"
2022-07-03T15:09:16.963 INFO:teuthology.orchestra.run.smithi195.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-07-03T15:09:16.964 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-07-03T15:09:16.964 INFO:teuthology.orchestra.run.smithi195.stdout:
2022-07-03T15:09:16.964 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-07-03T15:09:16.964 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-07-03T15:09:16.964 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-07-03T15:09:16.965 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2022-07-03T15:09:16.965 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server
2022-07-03T15:09:17.082 INFO:teuthology.dispatcher.supervisor:Running job 6911823
2022-07-03T15:09:17.083 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/bin/teuthology -v --owner scheduled_nshelke@teuthology --archive /home/teuthworker/archive/nshelke-2022-07-03_13:44:58-fs:volumes-wip-nshelke-in-prog-clones-02-07-2022-distro-default-smithi/6911823 --name nshelke-2022-07-03_13:44:58-fs:volumes-wip-nshelke-in-prog-clones-02-07-2022-distro-default-smithi --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} -- /home/teuthworker/archive/nshelke-2022-07-03_13:44:58-fs:volumes-wip-nshelke-in-prog-clones-02-07-2022-distro-default-smithi/6911823/orig.config.yaml
2022-07-03T15:09:17.087 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nshelke-2022-07-03_13:44:58-fs:volumes-wip-nshelke-in-prog-clones-02-07-2022-distro-default-smithi/6911823
2022-07-03T15:09:17.088 INFO:teuthology.dispatcher.supervisor:Job PID: 17199
2022-07-03T15:09:17.089 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-07-04T03:10:13.599 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-07-04T03:10:13.787 INFO:teuthology.kill:Killing Pids: {17199}
2022-07-04T03:10:15.581 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2']
2022-07-04T03:10:15.583 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']
2022-07-04T03:10:15.584 INFO:teuthology.misc:Compressing logs...
2022-07-04T03:10:15.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com'
2022-07-04T03:10:16.890 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/dispatcher/supervisor.py", line 290, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/remote.py", line 509, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi157
2022-07-04T03:10:16.956 INFO:teuthology.kill:No teuthology processes running
2022-07-04T03:10:18.548 INFO:teuthology.kill:Nuking machines: ['smithi157', 'smithi195']
2022-07-04T03:10:20.168 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-07-04T03:10:20.170 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-07-04T03:10:20.171 INFO:teuthology.kill:2022-07-04 03:10:20,167.167 INFO:teuthology.nuke:targets:
2022-07-04T03:10:20.173 INFO:teuthology.kill:  smithi157.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCS+K3ziVtelqtIhI8yh4T77iSXee+NSe5qXDqhnRfjrlPn68kphTA1Xop7B+TI72um3uqne9EtuqlLQbfuY5/Br1Gm+zWTTgFM3WqpyDg+kpQI8c+6qdQZzt5+WZxM5A76DdaAVu9FrN5//k0g3FctsrthFObQm0OR6N/kBSXioYHmFgNqvIDWjsAVEGQOfxmjOufLn33FMf8WmqqdENzKZKZvrdPmgEd5unohtTBqqX9akZdm5rnx+XvfICj6zIFbWgnJQ/IxTZfvqeZYsFpH5mbLxaqTPVyW2WLCkD+OG2UOWDnZzqw3GHubfTfGQyjaCctblhbxolkagPpfojopfMo1UqDqxCDqKTrQMHWdnGoWEhAtxb0hjp5SqOxLvTjSJ0ObF9h3sm3Fd6tF8P5JMSUUgKgYYgEBkje9+DItPKjRYCTX9/u+Q+asc1TEsIrDpizVoVkpvGItwTG/zCJb3Lwyi8Cuca2Hd8r6FvRuEYILBURKNzYBWpJQqqBx8u8=
2022-07-04T03:10:20.174 INFO:teuthology.kill:  smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeVpKqSxjKsKTqfenjgDLbqvoAGEzsI5sbkjpGmID5x160dliYtUkG1tKWdT1lA2/9Zf683TaOiCL2NSltRHNQzoSPn3b3ODgEUqCzQ6z6X1zko3AyylDgPWQxo+8050g45Cq1Y37Qfhx7lc/svAqrO7nd4e6ZoUI2fufp61vtShtohOtxoiFC0A0gs+WhNoSVxNgXUWgvmetdU2zNWLS3K/+evQXjzfKvD/1v59tjhUaJ4qIT5JdnKt6nct1W+wa8pELelsbNZ+C4MFDygOXDjKtuX5n/AYCOqL5Qe3lOqzXtP0LMEkWMANqpyR/Qby6SSSerS/cMm0tRIDb+W+2nTWwKmg2uPE4OwTIDGH6yIjZoFOA4zYzirh7+A2Mo/6887YrnvrFs4sBwe9Jln8qj7zbIcq5PcUoh2k0F0GI/y4r8bNPfQTs86HcJoi4SrOk4CPt2t6PE7URCHtSuxI7ADEteCeh9VJik3CYb46SS/WUDJ7/UEBEF83oCrHxs+iE=
2022-07-04T03:10:20.174 INFO:teuthology.kill:2022-07-04 03:10:20,168.168 INFO:teuthology.nuke:Checking targets against current locks
2022-07-04T03:10:20.258 INFO:teuthology.kill:2022-07-04 03:10:20,258.258 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-07-04T03:10:20.271 INFO:teuthology.kill:2022-07-04 03:10:20,271.271 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-07-04T03:10:20.361 INFO:teuthology.kill:2022-07-04 03:10:20,360.360 INFO:teuthology.orchestra.console:Power off smithi157
2022-07-04T03:10:20.414 INFO:teuthology.kill:2022-07-04 03:10:20,414.414 INFO:teuthology.orchestra.console:Power off smithi195
2022-07-04T03:10:28.798 INFO:teuthology.kill:2022-07-04 03:10:28,798.798 INFO:teuthology.orchestra.console:Power off for smithi195 completed
2022-07-04T03:10:28.963 INFO:teuthology.kill:2022-07-04 03:10:28,961.961 INFO:teuthology.lock.ops:unlocked: smithi195.front.sepia.ceph.com
2022-07-04T03:10:32.880 INFO:teuthology.kill:2022-07-04 03:10:32,879.879 INFO:teuthology.orchestra.console:Power off for smithi157 completed
2022-07-04T03:10:33.043 INFO:teuthology.kill:2022-07-04 03:10:33,043.043 INFO:teuthology.lock.ops:unlocked: smithi157.front.sepia.ceph.com
2022-07-04T03:12:33.432 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-07-04T03:12:33.547 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi157 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-07-03_03:31:04-rados-pacific-distro-default-smithi/6911117
2022-07-04T03:12:33.548 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi195 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-07-03_03:31:04-rados-pacific-distro-default-smithi/6911116