2022-07-03T03:13:47.139 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3
2022-07-03T03:13:47.279 INFO:teuthology.lock.ops:Start node 'smithi029.front.sepia.ceph.com' reimaging
2022-07-03T03:13:47.281 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: reset os type and version on server
2022-07-03T03:13:47.281 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2022-07-03T03:13:47.306 INFO:teuthology.lock.ops:Node 'smithi029.front.sepia.ceph.com' reimaging is complete
2022-07-03T03:13:47.306 INFO:teuthology.lock.ops:Start node 'smithi104.front.sepia.ceph.com' reimaging
2022-07-03T03:13:47.306 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: reset os type and version on server
2022-07-03T03:13:47.307 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2022-07-03T03:13:47.328 INFO:teuthology.lock.ops:Node 'smithi104.front.sepia.ceph.com' reimaging is complete
2022-07-03T03:13:47.518 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of centos 8.stream
2022-07-03T03:13:47.539 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of centos 8.stream
2022-07-03T03:13:47.844 INFO:teuthology.orchestra.console:Power off smithi029
2022-07-03T03:13:47.861 INFO:teuthology.orchestra.console:Power off smithi104
2022-07-03T03:13:47.991 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2022-07-03T03:13:48.008 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2022-07-03T03:13:48.093 INFO:teuthology.orchestra.console:Power on smithi029
2022-07-03T03:13:48.110 INFO:teuthology.orchestra.console:Power on smithi104
2022-07-03T03:13:52.358 INFO:teuthology.orchestra.console:Power on for smithi104 completed
2022-07-03T03:13:52.459 INFO:teuthology.provision.fog.smithi104:Waiting for deploy to finish
2022-07-03T03:13:56.573 INFO:teuthology.orchestra.console:Power on for smithi029 completed
2022-07-03T03:13:56.674 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish
2022-07-03T03:16:23.577 ERROR:teuthology.orchestra.connection:Error authenticating with smithi104.front.sepia.ceph.com: Authentication failed.
2022-07-03T03:18:04.343 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-07-03T03:18:29.633 INFO:teuthology.provision.fog.smithi104:Node is ready
2022-07-03T03:18:29.656 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104.front.sepia.ceph.com
2022-07-03T03:18:29.715 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104 smithi104.front.sepia.ceph.com smithi104
2022-07-03T03:18:30.102 INFO:teuthology.provision.fog.smithi104:Deploy complete!
2022-07-03T03:18:46.661 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-07-03T03:18:46.993 INFO:teuthology.provision.fog.smithi029:Node is ready
2022-07-03T03:18:47.014 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029.front.sepia.ceph.com
2022-07-03T03:18:47.074 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29 smithi029.front.sepia.ceph.com smithi029
2022-07-03T03:18:47.504 INFO:teuthology.provision.fog.smithi029:Deploy complete!
2022-07-03T03:18:47.673 INFO:teuthology.lock.ops:Checking smithi029.front.sepia.ceph.com
2022-07-03T03:18:47.674 INFO:teuthology.lock.ops:New key found. Updating...
2022-07-03T03:18:47.696 INFO:teuthology.lock.ops:Checking smithi104.front.sepia.ceph.com
2022-07-03T03:18:47.696 INFO:teuthology.lock.ops:New key found. Updating...
2022-07-03T03:18:47.716 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: set os type and version on server
2022-07-03T03:18:47.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2022-07-03T03:18:47.935 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64
2022-07-03T03:18:47.993 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream"
2022-07-03T03:18:47.994 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8"
2022-07-03T03:18:47.994 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos"
2022-07-03T03:18:47.994 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora"
2022-07-03T03:18:47.994 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8"
2022-07-03T03:18:47.995 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8"
2022-07-03T03:18:47.995 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 8"
2022-07-03T03:18:47.995 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31"
2022-07-03T03:18:47.995 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-07-03T03:18:47.995 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/"
2022-07-03T03:18:47.996 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-07-03T03:18:47.996 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-07-03T03:18:47.996 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-07-03T03:18:47.997 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2022-07-03T03:18:48.019 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: set os type and version on server
2022-07-03T03:18:48.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com'
2022-07-03T03:18:48.227 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64
2022-07-03T03:18:48.286 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream"
2022-07-03T03:18:48.287 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8"
2022-07-03T03:18:48.287 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos"
2022-07-03T03:18:48.287 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora"
2022-07-03T03:18:48.288 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8"
2022-07-03T03:18:48.288 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8"
2022-07-03T03:18:48.288 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 8"
2022-07-03T03:18:48.288 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31"
2022-07-03T03:18:48.289 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-07-03T03:18:48.289 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/"
2022-07-03T03:18:48.289 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-07-03T03:18:48.290 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-07-03T03:18:48.290 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-07-03T03:18:48.290 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2022-07-03T03:18:48.397 INFO:teuthology.dispatcher.supervisor:Running job 6910755
2022-07-03T03:18:48.398 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/bin/teuthology -v --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910755 --name rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} -- /home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910755/orig.config.yaml
2022-07-03T03:18:48.404 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910755
2022-07-03T03:18:48.404 INFO:teuthology.dispatcher.supervisor:Job PID: 29532
2022-07-03T03:18:48.405 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-07-03T15:19:12.548 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-07-03T15:19:12.740 INFO:teuthology.kill:Killing Pids: {29532}
2022-07-03T15:19:14.303 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-07-03T15:19:14.304 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-07-03T15:19:14.304 INFO:teuthology.misc:Compressing logs...
2022-07-03T15:19:14.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2022-07-03T15:19:15.834 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/dispatcher/supervisor.py", line 290, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/remote.py", line 509, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi029
2022-07-03T15:19:15.923 INFO:teuthology.kill:No teuthology processes running
2022-07-03T15:19:17.396 INFO:teuthology.kill:Nuking machines: ['smithi029', 'smithi104']
2022-07-03T15:19:18.564 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-07-03T15:19:18.565 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-07-03T15:19:18.567 INFO:teuthology.kill:2022-07-03 15:19:18,564.564 INFO:teuthology.nuke:targets:
2022-07-03T15:19:18.567 INFO:teuthology.kill:  smithi029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJSqlZBA1v9mdVQdbv9LSDoSlwYeTvleVo1Ti2hTtCjQhNRf8QRNJUECC3ejJVMIkzixpV2IKxPy4PloKCMEB94fmlfXytcRBt1bL+Xd70EMzPSH5+1y26/0Mb4gdRz17pj7U/7iTrrJTc3KWXVQ5+pO6hoZ+3FzcAgjd/3YM5TfMNN9LJSW8/u+TBDjxHzfU+pM3ciXTTeyHEHEj3iDe+kl1eDC5mrT5vuCPBgRZtqeqJ4rFhKY9z8qfp+6zamKye31tp9cImCZUGG7qotl/1Ybp7N1BDJPgQ8ohKElKfyUa7X0ovOiyHDJiN6C5JdRmiaOui8uVHgS8AnrAkOb1M2OMO8/O4Ymveza78+vB0XHMEIE+V4iiM6NTLdG2GlSANsnMdbGXNrMEk7l2UZKtWBx8uflIEeqOKyzsGLWdPQOnEuX4IclUvHh/9B2snKmXRIt+ULuv+chdKEh7Dr+NlL8j1yQCWJlso5VdRnZbWOF+n++aGL4mrdA1nSROIY7U=
2022-07-03T15:19:18.568 INFO:teuthology.kill:  smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEMKpXQh/1rpgoXFoGDvJnxKBbKiORqJdKRgb6jPNF1DLmQcwmldAm5MNNRRDr+U3nLF9hoxWCg+vefymWY2s8ynKc62/ZeA5vfmWgPbOh6k+TYuinj1AMIw7qQi2mqEYzmf0SF04EamkdPXdIIz732i7sJKYEG3FITiF+sQg06wnUEwDeIb9PuBHZqz1LRWSJ+aGRtpjeXPqLRYSmRa2ynXYw8xQasxJ7QTvE0OCOs9gRj78wPjUStV1l0WyqlOl93sqJtTPcfZdGdvX85IEJV9lHQ8HQe4wrkXqdo927bs+e2bPx0kHrhb0wEY4OpwVLQ5rAPe/SN4ulXO9reWEMV5+zP2rUKvs5IFsXeYwVY3njEZk4GfGpmF5heEn7jemyRer7fP2u3OvIMzVnnH1nSyyOlxx+pkMm14A0ulbcJtAS3OvCmCQ2KcyiOO26FIXxpZem6Tgqk6gZza0K/In5wXHEIRCtKg+ZvuDbGWyUNpKoxYj2qStRdLe81NPH7IE=
2022-07-03T15:19:18.568 INFO:teuthology.kill:2022-07-03 15:19:18,564.564 INFO:teuthology.nuke:Checking targets against current locks
2022-07-03T15:19:18.626 INFO:teuthology.kill:2022-07-03 15:19:18,626.626 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-07-03T15:19:18.658 INFO:teuthology.kill:2022-07-03 15:19:18,658.658 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-07-03T15:19:18.707 INFO:teuthology.kill:2022-07-03 15:19:18,706.706 INFO:teuthology.orchestra.console:Power off smithi029
2022-07-03T15:19:18.749 INFO:teuthology.kill:2022-07-03 15:19:18,748.748 INFO:teuthology.orchestra.console:Power off smithi104
2022-07-03T15:19:27.081 INFO:teuthology.kill:2022-07-03 15:19:27,081.081 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2022-07-03T15:19:27.245 INFO:teuthology.kill:2022-07-03 15:19:27,244.244 INFO:teuthology.lock.ops:unlocked: smithi029.front.sepia.ceph.com
2022-07-03T15:19:31.225 INFO:teuthology.kill:2022-07-03 15:19:31,224.224 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2022-07-03T15:19:31.377 INFO:teuthology.kill:2022-07-03 15:19:31,376.376 INFO:teuthology.lock.ops:unlocked: smithi104.front.sepia.ceph.com
2022-07-03T15:21:31.634 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-07-03T15:21:31.762 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi104 but it was locked by another job: /home/teuthworker/archive/rishabh-2022-07-03_15:10:28-fs-wip-rishabh-testing-20220702-1631-distro-default-smithi/6911870