2023-05-22T02:11:55.162 INFO:root:teuthology version: 0.0.1.dev101+gc6e0984
2023-05-22T02:12:20.377 INFO:teuthology.lock.ops:Start node 'smithi176.front.sepia.ceph.com' reimaging
2023-05-22T02:12:20.379 INFO:teuthology.lock.ops:Updating [smithi176.front.sepia.ceph.com]: reset os type and version on server
2023-05-22T02:12:20.379 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server
2023-05-22T02:12:25.414 INFO:teuthology.lock.ops:Node 'smithi176.front.sepia.ceph.com' reimaging is complete
2023-05-22T02:12:25.415 INFO:teuthology.lock.ops:Start node 'smithi190.front.sepia.ceph.com' reimaging
2023-05-22T02:12:25.415 INFO:teuthology.lock.ops:Updating [smithi190.front.sepia.ceph.com]: reset os type and version on server
2023-05-22T02:12:25.415 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server
2023-05-22T02:12:35.454 INFO:teuthology.lock.ops:Node 'smithi190.front.sepia.ceph.com' reimaging is complete
2023-05-22T02:13:00.766 INFO:teuthology.provision.fog.smithi176:Scheduling deploy of centos 8.stream
2023-05-22T02:13:00.808 INFO:teuthology.provision.fog.smithi190:Scheduling deploy of centos 8.stream
2023-05-22T02:13:09.752 INFO:teuthology.orchestra.console:Power off smithi176
2023-05-22T02:13:09.791 INFO:teuthology.orchestra.console:Power off smithi190
2023-05-22T02:13:25.141 INFO:teuthology.orchestra.console:Power off for smithi176 completed
2023-05-22T02:13:25.242 INFO:teuthology.orchestra.console:Power on smithi176
2023-05-22T02:13:29.263 INFO:teuthology.orchestra.console:Power off for smithi190 completed
2023-05-22T02:13:29.365 INFO:teuthology.orchestra.console:Power on smithi190
2023-05-22T02:13:29.499 INFO:teuthology.orchestra.console:Power on for smithi176 completed
2023-05-22T02:13:29.600 INFO:teuthology.provision.fog.smithi176:Waiting for deploy to finish
2023-05-22T02:13:34.458 INFO:teuthology.orchestra.console:Power on for smithi190 completed
2023-05-22T02:13:34.559 INFO:teuthology.provision.fog.smithi190:Waiting for deploy to finish
2023-05-22T02:16:12.752 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.176
2023-05-22T02:16:26.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.176
2023-05-22T02:16:28.016 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.190
2023-05-22T02:16:42.920 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.176
2023-05-22T02:16:46.004 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.190
2023-05-22T02:16:58.942 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.176
2023-05-22T02:17:02.000 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.190
2023-05-22T02:17:08.016 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.176
2023-05-22T02:18:08.072 WARNING:teuthology.provision.fog:timed out
2023-05-22T02:18:14.028 WARNING:teuthology.provision.fog:timed out
2023-05-22T02:18:19.381 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-22T02:18:19.627 INFO:teuthology.provision.fog.smithi190:Node is ready
2023-05-22T02:18:19.687 INFO:teuthology.orchestra.run.smithi190.stdout:smithi190.front.sepia.ceph.com
2023-05-22T02:18:19.747 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.15.190 smithi190.front.sepia.ceph.com smithi190
2023-05-22T02:18:25.090 INFO:teuthology.provision.fog.smithi190:Deploy complete!
2023-05-22T02:18:25.344 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-22T02:18:25.595 INFO:teuthology.provision.fog.smithi176:Node is ready
2023-05-22T02:18:25.614 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176.front.sepia.ceph.com
2023-05-22T02:18:25.674 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.15.176 smithi176.front.sepia.ceph.com smithi176
2023-05-22T02:18:26.016 INFO:teuthology.provision.fog.smithi176:Deploy complete!
2023-05-22T02:18:55.496 INFO:teuthology.lock.ops:Checking smithi176.front.sepia.ceph.com
2023-05-22T02:18:55.549 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-22T02:18:55.576 INFO:teuthology.lock.ops:Checking smithi190.front.sepia.ceph.com
2023-05-22T02:18:55.576 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-22T02:18:55.594 INFO:teuthology.lock.ops:Updating [smithi176.front.sepia.ceph.com]: set os type and version on server
2023-05-22T02:18:55.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com'
2023-05-22T02:18:56.316 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64
2023-05-22T02:18:56.374 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream"
2023-05-22T02:18:56.374 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="8"
2023-05-22T02:18:56.375 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos"
2023-05-22T02:18:56.375 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora"
2023-05-22T02:18:56.376 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="8"
2023-05-22T02:18:56.376 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el8"
2023-05-22T02:18:56.376 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 8"
2023-05-22T02:18:56.377 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31"
2023-05-22T02:18:56.377 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-05-22T02:18:56.377 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/"
2023-05-22T02:18:56.377 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-22T02:18:56.377 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-22T02:18:56.377 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-05-22T02:18:56.379 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server
2023-05-22T02:18:56.414 INFO:teuthology.lock.ops:Updating [smithi190.front.sepia.ceph.com]: set os type and version on server
2023-05-22T02:18:56.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com'
2023-05-22T02:18:56.675 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64
2023-05-22T02:18:56.734 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream"
2023-05-22T02:18:56.735 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="8"
2023-05-22T02:18:56.735 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos"
2023-05-22T02:18:56.735 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora"
2023-05-22T02:18:56.735 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="8"
2023-05-22T02:18:56.735 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el8"
2023-05-22T02:18:56.735 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 8"
2023-05-22T02:18:56.736 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31"
2023-05-22T02:18:56.736 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-05-22T02:18:56.736 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/"
2023-05-22T02:18:56.736 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-22T02:18:56.736 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-22T02:18:56.736 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-05-22T02:18:56.737 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server
2023-05-22T02:18:56.934 INFO:teuthology.dispatcher.supervisor:Running job 7282202
2023-05-22T02:18:56.935 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2023-05-22_01:10:22-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7282202 --name pdonnell-2023-05-22_01:10:22-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} -- /home/teuthworker/archive/pdonnell-2023-05-22_01:10:22-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7282202/orig.config.yaml
2023-05-22T02:18:56.951 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2023-05-22_01:10:22-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7282202
2023-05-22T02:18:56.952 INFO:teuthology.dispatcher.supervisor:Job PID: 8571
2023-05-22T02:18:56.952 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-05-22T14:20:32.798 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-05-22T14:20:32.912 INFO:teuthology.kill:Killing Pids: {8571}
2023-05-22T14:20:43.237 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2']
2023-05-22T14:20:43.238 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']
2023-05-22T14:20:43.238 INFO:teuthology.misc:Compressing logs...
2023-05-22T14:20:43.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com'
2023-05-22T14:20:43.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com'
2023-05-22T14:20:43.550 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2023-05-22T14:20:43.550 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2023-05-22T14:20:43.550 INFO:teuthology.orchestra.run.smithi176.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2023-05-22T14:20:43.557 INFO:teuthology.orchestra.run.smithi176.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	 89.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2023-05-22T14:20:46.580 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/dispatcher/supervisor.py", line 313, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/dispatcher/supervisor.py", line 378, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/misc.py", line 1368, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_c6e0984e7e7716eedf6039f15c6a9d1043e0c5f2/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi190
2023-05-22T14:20:46.641 INFO:teuthology.kill:No teuthology processes running
2023-05-22T14:20:47.560 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-05-22T14:22:47.843 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-05-22T14:22:47.949 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi176 but it was locked by another job: /home/teuthworker/archive/adking-2023-05-22_14:09:29-orch:cephadm-wip-adk3-testing-2023-05-21-1607-quincy-distro-default-smithi/7282433