2022-04-03T00:16:25.866 INFO:root:teuthology version: 1.1.0-d877b023
2022-04-03T00:16:26.000 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi138.front.sepia.ceph.com' reimaging
2022-04-03T00:16:26.001 INFO:teuthology.lock.ops:Updating [ubuntu@smithi138.front.sepia.ceph.com]: reset os type and version on server
2022-04-03T00:16:26.001 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server
2022-04-03T00:16:26.022 INFO:teuthology.lock.ops:Node 'ubuntu@smithi138.front.sepia.ceph.com' reimaging is complete
2022-04-03T00:16:26.022 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging
2022-04-03T00:16:26.022 INFO:teuthology.lock.ops:Updating [ubuntu@smithi195.front.sepia.ceph.com]: reset os type and version on server
2022-04-03T00:16:26.023 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server
2022-04-03T00:16:26.043 INFO:teuthology.lock.ops:Node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging is complete
2022-04-03T00:16:26.238 INFO:teuthology.provision.fog.smithi138:Scheduling deploy of rhel 8.4
2022-04-03T00:16:26.245 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of rhel 8.4
2022-04-03T00:16:26.564 INFO:teuthology.orchestra.console:Power off smithi195
2022-04-03T00:16:26.578 INFO:teuthology.orchestra.console:Power off smithi138
2022-04-03T00:16:34.941 INFO:teuthology.orchestra.console:Power off for smithi195 completed
2022-04-03T00:16:35.042 INFO:teuthology.orchestra.console:Power on smithi195
2022-04-03T00:16:39.060 INFO:teuthology.orchestra.console:Power off for smithi138 completed
2022-04-03T00:16:39.161 INFO:teuthology.orchestra.console:Power on smithi138
2022-04-03T00:16:43.410 INFO:teuthology.orchestra.console:Power on for smithi195 completed
2022-04-03T00:16:43.512 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish
2022-04-03T00:16:47.529 INFO:teuthology.orchestra.console:Power on for smithi138 completed
2022-04-03T00:16:47.630 INFO:teuthology.provision.fog.smithi138:Waiting for deploy to finish
2022-04-03T00:21:01.991 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-03T00:21:15.377 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-03T00:21:15.611 INFO:teuthology.provision.fog.smithi195:Node is ready
2022-04-03T00:21:15.626 INFO:teuthology.orchestra.run.smithi195.stdout:smithi195.front.sepia.ceph.com
2022-04-03T00:21:15.681 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.15.195 smithi195.front.sepia.ceph.com smithi195
2022-04-03T00:21:15.993 INFO:teuthology.provision.fog.smithi195:Deploy complete!
2022-04-03T00:21:22.382 INFO:teuthology.provision.fog.smithi138:Node is ready
2022-04-03T00:21:22.402 INFO:teuthology.orchestra.run.smithi138.stdout:smithi138.front.sepia.ceph.com
2022-04-03T00:21:22.459 INFO:teuthology.orchestra.run.smithi138.stdout:172.21.15.138 smithi138.front.sepia.ceph.com smithi138
2022-04-03T00:21:22.818 INFO:teuthology.provision.fog.smithi138:Deploy complete!
2022-04-03T00:21:22.968 INFO:teuthology.lock.ops:Checking smithi138.front.sepia.ceph.com
2022-04-03T00:21:22.969 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-03T00:21:22.998 INFO:teuthology.lock.ops:Checking smithi195.front.sepia.ceph.com
2022-04-03T00:21:22.999 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-03T00:21:23.017 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: set os type and version on server
2022-04-03T00:21:23.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com'
2022-04-03T00:21:23.242 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64
2022-04-03T00:21:23.296 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="Red Hat Enterprise Linux"
2022-04-03T00:21:23.297 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8.4 (Ootpa)"
2022-04-03T00:21:23.297 INFO:teuthology.orchestra.run.smithi138.stdout:ID="rhel"
2022-04-03T00:21:23.297 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="fedora"
2022-04-03T00:21:23.298 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8.4"
2022-04-03T00:21:23.298 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8"
2022-04-03T00:21:23.298 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-04-03T00:21:23.298 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31"
2022-04-03T00:21:23.299 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-04-03T00:21:23.299 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.redhat.com/"
2022-04-03T00:21:23.299 INFO:teuthology.orchestra.run.smithi138.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-04-03T00:21:23.299 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-03T00:21:23.299 INFO:teuthology.orchestra.run.smithi138.stdout:
2022-04-03T00:21:23.300 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-03T00:21:23.300 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-04-03T00:21:23.300 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-04-03T00:21:23.300 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-04-03T00:21:23.302 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server
2022-04-03T00:21:23.319 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: set os type and version on server
2022-04-03T00:21:23.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com'
2022-04-03T00:21:23.553 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64
2022-04-03T00:21:23.608 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="Red Hat Enterprise Linux"
2022-04-03T00:21:23.608 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8.4 (Ootpa)"
2022-04-03T00:21:23.609 INFO:teuthology.orchestra.run.smithi195.stdout:ID="rhel"
2022-04-03T00:21:23.609 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="fedora"
2022-04-03T00:21:23.609 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8.4"
2022-04-03T00:21:23.609 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8"
2022-04-03T00:21:23.609 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-04-03T00:21:23.610 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31"
2022-04-03T00:21:23.610 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-04-03T00:21:23.610 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://www.redhat.com/"
2022-04-03T00:21:23.610 INFO:teuthology.orchestra.run.smithi195.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-04-03T00:21:23.611 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-03T00:21:23.611 INFO:teuthology.orchestra.run.smithi195.stdout:
2022-04-03T00:21:23.611 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-03T00:21:23.611 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-04-03T00:21:23.611 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-04-03T00:21:23.612 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-04-03T00:21:23.612 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server
2022-04-03T00:21:23.723 INFO:teuthology.dispatcher.supervisor:Running job 6774111
2022-04-03T00:21:23.724 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774111 --name vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} -- /home/teuthworker/archive/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774111/orig.config.yaml
2022-04-03T00:21:23.728 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774111
2022-04-03T00:21:23.728 INFO:teuthology.dispatcher.supervisor:Job PID: 19966
2022-04-03T00:21:23.729 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-04-03T06:51:52.191 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-04-03T06:51:52.368 INFO:teuthology.kill:Killing Pids: {19966}
2022-04-03T06:51:53.618 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-04-03T06:51:53.619 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-04-03T06:51:53.620 INFO:teuthology.misc:Compressing logs...
2022-04-03T06:51:53.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com'
2022-04-03T06:51:53.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com'
2022-04-04T04:50:40.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 175, in _raise_for_status
    node=self.hostname)
teuthology.exceptions.ConnectionLostError: SSH connection to smithi138 was lost: 'sudo find /var/lib/ceph/crash -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2022-04-04T04:50:40.502 INFO:teuthology.kill:No teuthology processes running
2022-04-04T04:50:41.803 INFO:teuthology.kill:Nuking machines: ['smithi138', 'smithi195']
2022-04-04T04:50:42.786 INFO:teuthology.kill:2022-04-04 04:50:42,786.786 INFO:teuthology.nuke:targets:
2022-04-04T04:50:42.787 INFO:teuthology.kill:  smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsQ/XTCML4Jg1fi1rRu/icD+fElWwzT8aDleoH/JOh+Ls2661FJ/y6Ewz9MO8ORTtdRgAP9+lDR+TgAlMRs55E0uymqNelvUCeWFlbz4neHbx3K2/Le6NWZ/Qv8GChqvVU4CwT3JIBY+HFOVmGmugZXb8MZW7xRCrgTWcdEs8tye0vojE58dNrZIzkS26oTgCx8DoIRsRxxBnMHxOupoh3iXrshsM0hQ/wkra/5IIiy3PWWM+HOslJjmi0njdDgItf/Idom5PzKFjJFJFyNNXBzZHDnpIfBh8nuqEiVA68rADxF4ChYGH8iunH2yXsH6puHDtq4RGBW/SpcY6mrEyMldXzPysYoJvMaNnRfAYVsQe0rd3jGF7C0Z1rVjIlqxs+EhZCBMgPlHFSHIdwlktkJDTtG4OJ9r+6ycJvRdCy5C0aIB8XPgKWLs3RFGBut9/+DXriXzL6VKYPWfaJlcG606dpxCFTN9FbjKZI2eaB/6DgamIDb4hPsWaFZ3Of800=
2022-04-04T04:50:42.788 INFO:teuthology.kill:  smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBN82mIzXZVhxGjWKo7omnL+10xKA2sGXZT6+hM5eabtfMQp7TIzmPoshF106osfl/pYtPpSnbOUlFfwOUC/xSerUCRwpG4PAnuyFi8kq1a0My1AU33rrSuCGUXxWCRaa4X9R46vjq5WnDWYcAhHKBZ9UDVBm4k7+bxYlQ7xqep2kc4Hb/rhXtM+wshYdmRxmTYWYeIsVAKUXQ/pZ58HkxbweupaHcXlO8g/ZP0K+f+S+UDxzJZpY9MZs2aIAGeiYmHcfvGY4XXi+n6SOBVmgFhJXRM+hiEqAIWMeq4H4kYBpmfddAFwT4KxXVIGoJnHXaH8HyGcFxxPQNQm6oS7VoWoKFEpQzHybb7qGEFwa2xM89crdWsnrqiDi9kTRMyzOQJguKxabWflRlJXNztsd0xfCxgJ/AvT208ds+q1EF6EZDSIYGBYdRw5iDcIw7+x2+ywPM0n+WEEUEKVw02eRtihJgLu8HfZhflYtH/sSpZbCGE2nMS8M5aU47C2iyD58=
2022-04-04T04:50:42.812 INFO:teuthology.kill:2022-04-04 04:50:42,812.812 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-04T04:50:42.817 INFO:teuthology.kill:2022-04-04 04:50:42,817.817 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-04T04:50:42.899 INFO:teuthology.kill:2022-04-04 04:50:42,899.899 INFO:teuthology.orchestra.console:Power off smithi138
2022-04-04T04:50:42.919 INFO:teuthology.kill:2022-04-04 04:50:42,918.918 INFO:teuthology.orchestra.console:Power off smithi195
2022-04-04T04:50:47.172 INFO:teuthology.kill:2022-04-04 04:50:47,172.172 INFO:teuthology.orchestra.console:Power off for smithi138 completed
2022-04-04T04:50:47.184 INFO:teuthology.kill:2022-04-04 04:50:47,184.184 INFO:teuthology.orchestra.console:Power off for smithi195 completed
2022-04-04T04:51:14.475 INFO:teuthology.kill:Traceback (most recent call last):
2022-04-04T04:51:14.475 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology-nuke", line 33, in <module>
2022-04-04T04:51:14.476 INFO:teuthology.kill:    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-nuke')())
2022-04-04T04:51:14.476 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/scripts/nuke.py", line 47, in main
2022-04-04T04:51:14.476 INFO:teuthology.kill:    teuthology.nuke.main(args)
2022-04-04T04:51:14.477 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/nuke/__init__.py", line 235, in main
2022-04-04T04:51:14.477 INFO:teuthology.kill:    nuke(ctx, ctx.unlock, ctx.synch_clocks, ctx.noipmi, ctx.keep_logs, not ctx.no_reboot)
2022-04-04T04:51:14.477 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/nuke/__init__.py", line 273, in nuke
2022-04-04T04:51:14.477 INFO:teuthology.kill:    for unnuked in p:
2022-04-04T04:51:14.478 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 98, in __next__
2022-04-04T04:51:14.478 INFO:teuthology.kill:    resurrect_traceback(result)
2022-04-04T04:51:14.478 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 30, in resurrect_traceback
2022-04-04T04:51:14.479 INFO:teuthology.kill:    raise exc.exc_info[1]
2022-04-04T04:51:14.479 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 23, in capture_traceback
2022-04-04T04:51:14.479 INFO:teuthology.kill:    return func(*args, **kwargs)
2022-04-04T04:51:14.479 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/nuke/__init__.py", line 304, in nuke_one
2022-04-04T04:51:14.480 INFO:teuthology.kill:    unlock_one(ctx, list(target.keys())[0], ctx.owner)
2022-04-04T04:51:14.480 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/lock/ops.py", line 209, in unlock_one
2022-04-04T04:51:14.480 INFO:teuthology.kill:    while proceed():
2022-04-04T04:51:14.480 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 133, in __call__
2022-04-04T04:51:14.481 INFO:teuthology.kill:    raise MaxWhileTries(error_msg)
2022-04-04T04:51:14.481 INFO:teuthology.kill:teuthology.exceptions.MaxWhileTries: 'unlock smithi138.front.sepia.ceph.com' reached maximum tries (10) after waiting for 32.5 seconds
2022-04-04T04:53:14.844 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-04-04T04:53:14.950 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi138 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-04-01_03:20:03-krbd-quincy-testing-default-smithi/6771506
2022-04-04T04:53:14.950 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi195 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-04-01_03:20:03-krbd-quincy-testing-default-smithi/6771506