2022-04-03T21:33:25.496 INFO:root:teuthology version: 1.1.0-d877b023
2022-04-03T21:33:25.630 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi049.front.sepia.ceph.com' reimaging
2022-04-03T21:33:25.631 INFO:teuthology.lock.ops:Updating [ubuntu@smithi049.front.sepia.ceph.com]: reset os type and version on server
2022-04-03T21:33:25.632 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server
2022-04-03T21:33:25.653 INFO:teuthology.lock.ops:Node 'ubuntu@smithi049.front.sepia.ceph.com' reimaging is complete
2022-04-03T21:33:25.654 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi117.front.sepia.ceph.com' reimaging
2022-04-03T21:33:25.654 INFO:teuthology.lock.ops:Updating [ubuntu@smithi117.front.sepia.ceph.com]: reset os type and version on server
2022-04-03T21:33:25.654 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server
2022-04-03T21:33:25.675 INFO:teuthology.lock.ops:Node 'ubuntu@smithi117.front.sepia.ceph.com' reimaging is complete
2022-04-03T21:33:25.854 INFO:teuthology.provision.fog.smithi049:Scheduling deploy of centos 8.stream
2022-04-03T21:33:25.873 INFO:teuthology.provision.fog.smithi117:Scheduling deploy of centos 8.stream
2022-04-03T21:33:26.194 INFO:teuthology.orchestra.console:Power off smithi049
2022-04-03T21:33:26.210 INFO:teuthology.orchestra.console:Power off smithi117
2022-04-03T21:33:34.572 INFO:teuthology.orchestra.console:Power off for smithi049 completed
2022-04-03T21:33:34.578 INFO:teuthology.orchestra.console:Power off for smithi117 completed
2022-04-03T21:33:34.674 INFO:teuthology.orchestra.console:Power on smithi049
2022-04-03T21:33:34.681 INFO:teuthology.orchestra.console:Power on smithi117
2022-04-03T21:33:43.037 INFO:teuthology.orchestra.console:Power on for smithi049 completed
2022-04-03T21:33:43.042 INFO:teuthology.orchestra.console:Power on for smithi117 completed
2022-04-03T21:33:43.139 INFO:teuthology.provision.fog.smithi049:Waiting for deploy to finish
2022-04-03T21:33:43.143 INFO:teuthology.provision.fog.smithi117:Waiting for deploy to finish
2022-04-03T21:38:12.175 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-03T21:38:12.395 INFO:teuthology.provision.fog.smithi117:Node is ready
2022-04-03T21:38:12.413 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117.front.sepia.ceph.com
2022-04-03T21:38:12.469 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117 smithi117.front.sepia.ceph.com smithi117
2022-04-03T21:38:12.828 INFO:teuthology.provision.fog.smithi117:Deploy complete!
2022-04-03T21:39:09.529 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-04-03T21:39:09.740 INFO:teuthology.provision.fog.smithi049:Node is ready
2022-04-03T21:39:09.800 INFO:teuthology.orchestra.run.smithi049.stdout:smithi049.front.sepia.ceph.com
2022-04-03T21:39:09.859 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49 smithi049.front.sepia.ceph.com smithi049
2022-04-03T21:39:10.229 INFO:teuthology.provision.fog.smithi049:Deploy complete!
2022-04-03T21:39:10.395 INFO:teuthology.lock.ops:Checking smithi049.front.sepia.ceph.com
2022-04-03T21:39:10.396 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-03T21:39:10.415 INFO:teuthology.lock.ops:Checking smithi117.front.sepia.ceph.com
2022-04-03T21:39:10.415 INFO:teuthology.lock.ops:New key found. Updating...
2022-04-03T21:39:10.435 INFO:teuthology.lock.ops:Updating [smithi049.front.sepia.ceph.com]: set os type and version on server
2022-04-03T21:39:10.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com'
2022-04-03T21:39:10.624 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64
2022-04-03T21:39:10.681 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream"
2022-04-03T21:39:10.683 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="8"
2022-04-03T21:39:10.683 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos"
2022-04-03T21:39:10.683 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora"
2022-04-03T21:39:10.683 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="8"
2022-04-03T21:39:10.684 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el8"
2022-04-03T21:39:10.684 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-03T21:39:10.684 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31"
2022-04-03T21:39:10.684 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-03T21:39:10.685 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/"
2022-04-03T21:39:10.685 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-03T21:39:10.685 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-03T21:39:10.685 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-03T21:39:10.687 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server
2022-04-03T21:39:10.705 INFO:teuthology.lock.ops:Updating [smithi117.front.sepia.ceph.com]: set os type and version on server
2022-04-03T21:39:10.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com'
2022-04-03T21:39:10.950 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64
2022-04-03T21:39:11.008 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream"
2022-04-03T21:39:11.008 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="8"
2022-04-03T21:39:11.008 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos"
2022-04-03T21:39:11.009 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora"
2022-04-03T21:39:11.009 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="8"
2022-04-03T21:39:11.009 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el8"
2022-04-03T21:39:11.009 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 8"
2022-04-03T21:39:11.009 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31"
2022-04-03T21:39:11.010 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-04-03T21:39:11.010 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/"
2022-04-03T21:39:11.010 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-04-03T21:39:11.010 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-04-03T21:39:11.011 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-04-03T21:39:11.011 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server
2022-04-03T21:39:11.109 INFO:teuthology.dispatcher.supervisor:Running job 6775668
2022-04-03T21:39:11.109 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6775668 --name vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi --description fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} -- /home/teuthworker/archive/vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6775668/orig.config.yaml
2022-04-03T21:39:11.112 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6775668
2022-04-03T21:39:11.113 INFO:teuthology.dispatcher.supervisor:Job PID: 16706
2022-04-03T21:39:11.114 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-04-04T04:09:38.697 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-04-04T04:09:38.864 INFO:teuthology.kill:Killing Pids: {16706}
2022-04-04T04:09:39.984 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-04-04T04:09:39.985 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-04-04T04:09:39.985 INFO:teuthology.misc:Compressing logs...
2022-04-04T04:09:39.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com'
2022-04-04T04:09:40.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com'
2022-04-04T05:00:53.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 175, in _raise_for_status
    node=self.hostname)
teuthology.exceptions.ConnectionLostError: SSH connection to smithi049 was lost: 'sudo find /var/lib/ceph/crash -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2022-04-04T05:00:53.212 INFO:teuthology.kill:No teuthology processes running
2022-04-04T05:00:54.396 INFO:teuthology.kill:Nuking machines: ['smithi049', 'smithi117']
2022-04-04T05:00:55.360 INFO:teuthology.kill:2022-04-04 05:00:55,360.360 INFO:teuthology.nuke:targets:
2022-04-04T05:00:55.361 INFO:teuthology.kill:  smithi049.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcx6FNUI1+8IhYWPjRR05YLwLbzlNwRkh+BjacmtArX8/l9VlODyae8zxqgcfE035IF/jZSJ08cifu11EDuH9Pb5bgHyZd8riLhOddNZioOXq3eswWIl3zklgt8pS2wXL//pv9L/HfUz20qJGAmzqL2e+9boYQApSn5e34pxEnJ3pYVlzsCs0XO7MeaRociyEEr0NpITMGZ65xzvjKCt+abPO1xUuQ8Vvk8+oJ8sUvFZ7qfb5ooS8v/qRlXZ9asRdjxsXQktA7Q6LbTYTnZJk0w9EfgQM3qFjkf6Mo8xw6hqLYbYZyZ7SjfxJqVqO73KQVFj40zblgLUfpYkuI3v+QgkWqHhsFFLvMCRqRC+aRqaCwwvCz1mefZGzng1fmPNTt7ohGGciS7hhQqyEkP5tqmdr/cMmzueru4IG/tsScKE0kDUP+FygdUDyc5x0Ie+HFRJY/ENit1bULESlx9OpxV1K42qN0IYGg92s9E4Etx6bHBJahtj7pYz1bAyKjpcc=
2022-04-04T05:00:55.361 INFO:teuthology.kill:  smithi117.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2n0KUnOcrabfO+SDulzmks2AdtqPx78dAGzF/gfk0twvbCAsea+9vnH4TVfejC8NLTQOR7UyvjDdlrVuiDPCqY1BgcLw+8IzgbgaM7UdrzSly181ASQMoBA6nMafWAkYzivZyv3VeahBcoPvev7UU/hgPPj8rqTPtnbYdjovgeXtkatBJdzxYPUae8rebYHpTnxzi4OQBoLXcBLGI04MngEIWyiB3k9vQHMbB7ZKpFvgdq7lh9qfF3085NuRt8OCjO5M3+nzYhAga9NGNOhnRUF1D5jJiq4NEf6RZWRtTPjzoa6wCK9nu4P2Qdv+M/QzeQLqwh7Q41mGj/PXYOEcY6DhMsu3V+VaMsURMGYBEzuZVTXHJRxsOyIIJiNc/lvFZMLr/La6REUJqqWgeCkpmp1CJ4036Ds6hbtYEzD8DEBG/KG4fpRkudoU+kBcQh49kg/x6Te965erwRIvy0bNUoTZZD4UnrWqiYVs6nYwxixprFvMC/5O0Hm7nTHc/cMc=
2022-04-04T05:00:55.390 INFO:teuthology.kill:2022-04-04 05:00:55,389.389 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-04T05:00:55.391 INFO:teuthology.kill:2022-04-04 05:00:55,390.390 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-04-04T05:00:55.478 INFO:teuthology.kill:2022-04-04 05:00:55,478.478 INFO:teuthology.orchestra.console:Power off smithi049
2022-04-04T05:00:55.492 INFO:teuthology.kill:2022-04-04 05:00:55,492.492 INFO:teuthology.orchestra.console:Power off smithi117
2022-04-04T05:00:59.736 INFO:teuthology.kill:2022-04-04 05:00:59,736.736 INFO:teuthology.orchestra.console:Power off for smithi049 completed
2022-04-04T05:00:59.743 INFO:teuthology.kill:2022-04-04 05:00:59,743.743 INFO:teuthology.orchestra.console:Power off for smithi117 completed
2022-04-04T05:00:59.881 INFO:teuthology.kill:2022-04-04 05:00:59,880.880 INFO:teuthology.lock.ops:unlocked: smithi049.front.sepia.ceph.com
2022-04-04T05:01:27.023 INFO:teuthology.kill:Traceback (most recent call last):
2022-04-04T05:01:27.023 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology-nuke", line 33, in <module>
2022-04-04T05:01:27.024 INFO:teuthology.kill:    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-nuke')())
2022-04-04T05:01:27.024 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/scripts/nuke.py", line 47, in main
2022-04-04T05:01:27.024 INFO:teuthology.kill:    teuthology.nuke.main(args)
2022-04-04T05:01:27.025 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/nuke/__init__.py", line 235, in main
2022-04-04T05:01:27.025 INFO:teuthology.kill:    nuke(ctx, ctx.unlock, ctx.synch_clocks, ctx.noipmi, ctx.keep_logs, not ctx.no_reboot)
2022-04-04T05:01:27.025 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/nuke/__init__.py", line 273, in nuke
2022-04-04T05:01:27.025 INFO:teuthology.kill:    for unnuked in p:
2022-04-04T05:01:27.026 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 98, in __next__
2022-04-04T05:01:27.026 INFO:teuthology.kill:    resurrect_traceback(result)
2022-04-04T05:01:27.026 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 30, in resurrect_traceback
2022-04-04T05:01:27.027 INFO:teuthology.kill:    raise exc.exc_info[1]
2022-04-04T05:01:27.027 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 23, in capture_traceback
2022-04-04T05:01:27.027 INFO:teuthology.kill:    return func(*args, **kwargs)
2022-04-04T05:01:27.027 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/nuke/__init__.py", line 304, in nuke_one
2022-04-04T05:01:27.028 INFO:teuthology.kill:    unlock_one(ctx, list(target.keys())[0], ctx.owner)
2022-04-04T05:01:27.028 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/lock/ops.py", line 209, in unlock_one
2022-04-04T05:01:27.028 INFO:teuthology.kill:    while proceed():
2022-04-04T05:01:27.028 INFO:teuthology.kill:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/contextutil.py", line 133, in __call__
2022-04-04T05:01:27.029 INFO:teuthology.kill:    raise MaxWhileTries(error_msg)
2022-04-04T05:01:27.029 INFO:teuthology.kill:teuthology.exceptions.MaxWhileTries: 'unlock smithi117.front.sepia.ceph.com' reached maximum tries (10) after waiting for 32.5 seconds
2022-04-04T05:03:27.369 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-04-04T05:03:27.476 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi049 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-04-01_03:20:03-krbd-quincy-testing-default-smithi/6771519
2022-04-04T05:03:27.476 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi117 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-04-01_03:20:03-krbd-quincy-testing-default-smithi/6771524