2022-04-02T22:21:34.353 INFO:root:teuthology version: 1.1.0-d877b023
2022-04-02T22:21:34.475 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging
2022-04-02T22:21:34.476 INFO:teuthology.lock.ops:Updating [ubuntu@smithi085.front.sepia.ceph.com]: reset os type and version on server
2022-04-02T22:21:34.476 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server
2022-04-02T22:21:34.494 INFO:teuthology.lock.ops:Node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging is complete
2022-04-02T22:21:34.495 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi097.front.sepia.ceph.com' reimaging
2022-04-02T22:21:34.495 INFO:teuthology.lock.ops:Updating [ubuntu@smithi097.front.sepia.ceph.com]: reset os type and version on server
2022-04-02T22:21:34.496 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server
2022-04-02T22:21:34.514 INFO:teuthology.lock.ops:Node 'ubuntu@smithi097.front.sepia.ceph.com' reimaging is complete
2022-04-02T22:21:34.711 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of rhel 8.4
2022-04-02T22:21:34.732 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of rhel 8.4
2022-04-02T22:21:35.071 INFO:teuthology.orchestra.console:Power off smithi085
2022-04-02T22:21:35.111 INFO:teuthology.orchestra.console:Power off smithi097
2022-04-02T22:21:35.221 INFO:teuthology.orchestra.console:Power off for smithi085 completed
2022-04-02T22:21:35.322 INFO:teuthology.orchestra.console:Power on smithi085
2022-04-02T22:21:39.572 INFO:teuthology.orchestra.console:Power on for smithi085 completed
2022-04-02T22:21:39.673 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish
2022-04-02T22:21:43.478 INFO:teuthology.orchestra.console:Power off for smithi097 completed
2022-04-02T22:21:43.580 INFO:teuthology.orchestra.console:Power on smithi097
2022-04-02T22:21:51.943 INFO:teuthology.orchestra.console:Power on for smithi097 completed
2022-04-02T22:21:52.044 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish
2022-04-02T22:25:58.907 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-02T22:26:18.000 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-02T22:26:18.230 INFO:teuthology.provision.fog.smithi085:Node is ready
2022-04-02T22:26:18.248 INFO:teuthology.orchestra.run.smithi085.stdout:smithi085.front.sepia.ceph.com
2022-04-02T22:26:18.304 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.85 smithi085.front.sepia.ceph.com smithi085
2022-04-02T22:26:18.631 INFO:teuthology.provision.fog.smithi085:Deploy complete!
2022-04-02T22:26:19.555 INFO:teuthology.provision.fog.smithi097:Node is ready
2022-04-02T22:26:19.575 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097.front.sepia.ceph.com
2022-04-02T22:26:19.631 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97 smithi097.front.sepia.ceph.com smithi097
2022-04-02T22:26:19.975 INFO:teuthology.provision.fog.smithi097:Deploy complete!
2022-04-02T22:26:20.157 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com
2022-04-02T22:26:20.158 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-02T22:26:20.183 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com
2022-04-02T22:26:20.183 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-02T22:26:20.202 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server
2022-04-02T22:26:20.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com'
2022-04-02T22:26:20.434 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64
2022-04-02T22:26:20.489 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="Red Hat Enterprise Linux"
2022-04-02T22:26:20.489 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="8.4 (Ootpa)"
2022-04-02T22:26:20.490 INFO:teuthology.orchestra.run.smithi085.stdout:ID="rhel"
2022-04-02T22:26:20.490 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE="fedora"
2022-04-02T22:26:20.490 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="8.4"
2022-04-02T22:26:20.490 INFO:teuthology.orchestra.run.smithi085.stdout:PLATFORM_ID="platform:el8"
2022-04-02T22:26:20.491 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-04-02T22:26:20.491 INFO:teuthology.orchestra.run.smithi085.stdout:ANSI_COLOR="0;31"
2022-04-02T22:26:20.491 INFO:teuthology.orchestra.run.smithi085.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-04-02T22:26:20.491 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://www.redhat.com/"
2022-04-02T22:26:20.492 INFO:teuthology.orchestra.run.smithi085.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-04-02T22:26:20.492 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-02T22:26:20.492 INFO:teuthology.orchestra.run.smithi085.stdout:
2022-04-02T22:26:20.492 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-02T22:26:20.492 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-04-02T22:26:20.493 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-04-02T22:26:20.493 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-04-02T22:26:20.495 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server
2022-04-02T22:26:20.514 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server
2022-04-02T22:26:20.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com'
2022-04-02T22:26:20.747 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64
2022-04-02T22:26:20.801 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="Red Hat Enterprise Linux"
2022-04-02T22:26:20.801 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8.4 (Ootpa)"
2022-04-02T22:26:20.802 INFO:teuthology.orchestra.run.smithi097.stdout:ID="rhel"
2022-04-02T22:26:20.802 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="fedora"
2022-04-02T22:26:20.802 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8.4"
2022-04-02T22:26:20.802 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8"
2022-04-02T22:26:20.803 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-04-02T22:26:20.803 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31"
2022-04-02T22:26:20.803 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-04-02T22:26:20.803 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.redhat.com/"
2022-04-02T22:26:20.803 INFO:teuthology.orchestra.run.smithi097.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-04-02T22:26:20.804 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-02T22:26:20.804 INFO:teuthology.orchestra.run.smithi097.stdout:
2022-04-02T22:26:20.804 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-02T22:26:20.804 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-04-02T22:26:20.805 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-04-02T22:26:20.805 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-04-02T22:26:20.805 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server
2022-04-02T22:26:20.908 INFO:teuthology.dispatcher.supervisor:Running job 6773984
2022-04-02T22:26:20.909 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773984 --name vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi --description fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} -- /home/teuthworker/archive/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773984/orig.config.yaml
2022-04-02T22:26:20.913 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773984
2022-04-02T22:26:20.913 INFO:teuthology.dispatcher.supervisor:Job PID: 21870
2022-04-02T22:26:20.914 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-04-03T04:56:49.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-04-03T04:56:49.809 INFO:teuthology.kill:Killing Pids: {21870}
2022-04-03T04:56:50.936 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-04-03T04:56:50.937 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2022-04-03T04:56:50.937 INFO:teuthology.misc:Compressing logs...
2022-04-03T04:56:50.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi085.front.sepia.ceph.com'
2022-04-03T04:56:51.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com'
2022-04-04T04:50:40.331 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 175, in _raise_for_status
    node=self.hostname)
teuthology.exceptions.ConnectionLostError: SSH connection to smithi085 was lost: 'sudo find /var/lib/ceph/crash -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2022-04-04T04:50:40.473 INFO:teuthology.kill:No teuthology processes running
2022-04-04T04:50:41.733 INFO:teuthology.kill:Nuking machines: ['smithi085', 'smithi097']
2022-04-04T04:50:42.757 INFO:teuthology.kill:2022-04-04 04:50:42,757.757 INFO:teuthology.nuke:targets:
2022-04-04T04:50:42.758 INFO:teuthology.kill:  smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxWFb18IZFETJNZsRkH1CpGHQpi3+9iK7J7cGCCCns1gLhfhbreQKOMPw4KvsLtt0Lx1Fs57Z7bBwtFxqBISlMAsMg8inMUzEYt3g4wvmgD5J1fC8K/aov6PK0Crv1uM5egLv65qNmKL8INuG43vg6We77E5EnribUTdGm9aDTZNTD6KgOTwnk8msxN9sg2q1tcD/b/S7vJUEGPixdzmY9IiKdV3QskBRd0zMVoAp2f3Ih1ZCvrB9xpeov2BIgzurJ9ob2LoT+zYArWj+RQZgBC4wvdfNaby+QhGF8IL9NbvHdVhJq2lU6RDxdPQOqHXt7IYFD4IH4nTPbW3s0LTx2Rwf+VwJNAMnkAWvtgSVYtGoptw2cXGlDwh0cs8vE5MZeR1UlU1NM4VBcUf/pDpPRa4jgktQzGdMeHEuoeQ3vVVxk0+QB5bWq/lDv1a5pLpcf0uvrEhGAV3oUzHjSICSwcudV+Oft3dLov/nWmATuxU26WyBwpG2P1q2rzfiDpb8=
2022-04-04T04:50:42.758 INFO:teuthology.kill:  smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDW5fkkCaqyw5DPlmFgplCXOpWcCeaVddvxe0fps4+rBSkZYf8IFCr2UJF9KM1vNRGVb6Iyl8vd2oZ+D7Xuvwd90MyUbBGX5htX8kpaLD0ymvUMwKTd49aN8i4wIEulmbVU1XESu8GGMVGwRkanuxkKnlX3N/oHKR2OuJ5/+2b57J8tIiG2b6R+Kg7YQnYEo+nCGj2e9TfuNj7WOQLN/bzsgnGh3oM12IS4OWLxaWVt6+rw1H5vcyAfZt1Wz1MN5FHxX4Z9tVV8tpYC9NpvKxvvkR4hm9WB9UOM4It8SoQKIc/uMdegsPu7QCgq9OwMFdkknq4XSFTxZHHuWUAFaNxEx7a0eql1O0dn2WpiI1KvFVDDwUJyuRyHhbLDz90RANl9UQM89/tvwxCMZo6doREi3Z2pF1qLUELHS577hj4GP6tFXVD0zSMZ/f/I1O2xYDHwMMIukS/W5Yt6gJIR1vUsTaB5AH6wu50SvaBQ8qLlQ4bCb8MZX980R/pDTMbzhFk=
2022-04-04T04:50:42.784 INFO:teuthology.kill:2022-04-04 04:50:42,784.784 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-04T04:50:42.789 INFO:teuthology.kill:2022-04-04 04:50:42,788.788 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-04T04:50:42.890 INFO:teuthology.kill:2022-04-04 04:50:42,890.890 INFO:teuthology.orchestra.console:Power off smithi085
2022-04-04T04:50:42.905 INFO:teuthology.kill:2022-04-04 04:50:42,905.905 INFO:teuthology.orchestra.console:Power off smithi097
2022-04-04T04:50:47.157 INFO:teuthology.kill:2022-04-04 04:50:47,156.156 INFO:teuthology.orchestra.console:Power off for smithi085 completed
2022-04-04T04:50:47.171 INFO:teuthology.kill:2022-04-04 04:50:47,171.171 INFO:teuthology.orchestra.console:Power off for smithi097 completed
2022-04-04T04:50:47.320 INFO:teuthology.kill:2022-04-04 04:50:47,319.319 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com
2022-04-04T04:51:14.441 INFO:teuthology.kill:Traceback (most recent call last):
2022-04-04T04:51:14.441 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology-nuke", line 33, in <module>
2022-04-04T04:51:14.442 INFO:teuthology.kill:    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-nuke')())
2022-04-04T04:51:14.442 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/scripts/nuke.py", line 47, in main
2022-04-04T04:51:14.442 INFO:teuthology.kill:    teuthology.nuke.main(args)
2022-04-04T04:51:14.443 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/nuke/__init__.py", line 235, in main
2022-04-04T04:51:14.443 INFO:teuthology.kill:    nuke(ctx, ctx.unlock, ctx.synch_clocks, ctx.noipmi, ctx.keep_logs, not ctx.no_reboot)
2022-04-04T04:51:14.443 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/nuke/__init__.py", line 273, in nuke
2022-04-04T04:51:14.443 INFO:teuthology.kill:    for unnuked in p:
2022-04-04T04:51:14.444 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 98, in __next__
2022-04-04T04:51:14.444 INFO:teuthology.kill:    resurrect_traceback(result)
2022-04-04T04:51:14.444 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 30, in resurrect_traceback
2022-04-04T04:51:14.444 INFO:teuthology.kill:    raise exc.exc_info[1]
2022-04-04T04:51:14.445 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 23, in capture_traceback
2022-04-04T04:51:14.445 INFO:teuthology.kill:    return func(*args, **kwargs)
2022-04-04T04:51:14.445 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/nuke/__init__.py", line 304, in nuke_one
2022-04-04T04:51:14.445 INFO:teuthology.kill:    unlock_one(ctx, list(target.keys())[0], ctx.owner)
2022-04-04T04:51:14.446 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/lock/ops.py", line 209, in unlock_one
2022-04-04T04:51:14.446 INFO:teuthology.kill:    while proceed():
2022-04-04T04:51:14.446 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 133, in __call__
2022-04-04T04:51:14.447 INFO:teuthology.kill:    raise MaxWhileTries(error_msg)
2022-04-04T04:51:14.447 INFO:teuthology.kill:teuthology.exceptions.MaxWhileTries: 'unlock smithi085.front.sepia.ceph.com' reached maximum tries (10) after waiting for 32.5 seconds
2022-04-04T04:53:14.782 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-04-04T04:53:14.899 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi085 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-04-01_03:20:03-krbd-quincy-testing-default-smithi/6771507
2022-04-04T04:53:14.900 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi097 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-04-01_03:20:03-krbd-quincy-testing-default-smithi/6771502