2022-03-13T04:17:41.781 INFO:root:teuthology version: 1.1.0-fbfc7b38
2022-03-13T04:17:41.903 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging
2022-03-13T04:17:41.904 INFO:teuthology.lock.ops:Updating [ubuntu@smithi029.front.sepia.ceph.com]: reset os type and version on server
2022-03-13T04:17:41.904 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2022-03-13T04:17:41.924 INFO:teuthology.lock.ops:Node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging is complete
2022-03-13T04:17:41.925 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging
2022-03-13T04:17:41.925 INFO:teuthology.lock.ops:Updating [ubuntu@smithi074.front.sepia.ceph.com]: reset os type and version on server
2022-03-13T04:17:41.925 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server
2022-03-13T04:17:41.945 INFO:teuthology.lock.ops:Node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging is complete
2022-03-13T04:17:42.148 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of centos 8.stream
2022-03-13T04:17:42.153 INFO:teuthology.provision.fog.smithi074:Scheduling deploy of centos 8.stream
2022-03-13T04:17:42.502 INFO:teuthology.orchestra.console:Power off smithi074
2022-03-13T04:17:42.563 INFO:teuthology.orchestra.console:Power off smithi029
2022-03-13T04:17:50.926 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2022-03-13T04:17:51.027 INFO:teuthology.orchestra.console:Power on smithi029
2022-03-13T04:17:54.983 INFO:teuthology.orchestra.console:Power off for smithi074 completed
2022-03-13T04:17:55.084 INFO:teuthology.orchestra.console:Power on smithi074
2022-03-13T04:17:59.386 INFO:teuthology.orchestra.console:Power on for smithi029 completed
2022-03-13T04:17:59.487 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish
2022-03-13T04:18:03.447 INFO:teuthology.orchestra.console:Power on for smithi074 completed
2022-03-13T04:18:03.548 INFO:teuthology.provision.fog.smithi074:Waiting for deploy to finish
2022-03-13T04:22:09.909 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-13T04:22:35.673 INFO:teuthology.provision.fog.smithi074:Node is ready
2022-03-13T04:22:35.696 INFO:teuthology.orchestra.run.smithi074.stdout:smithi074.front.sepia.ceph.com
2022-03-13T04:22:35.755 INFO:teuthology.orchestra.run.smithi074.stdout:172.21.15.74 smithi074.front.sepia.ceph.com smithi074
2022-03-13T04:22:36.120 INFO:teuthology.provision.fog.smithi074:Deploy complete!
2022-03-13T04:22:37.520 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-13T04:22:37.809 INFO:teuthology.provision.fog.smithi029:Node is ready
2022-03-13T04:22:37.828 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029.front.sepia.ceph.com
2022-03-13T04:22:37.886 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29 smithi029.front.sepia.ceph.com smithi029
2022-03-13T04:22:38.238 INFO:teuthology.provision.fog.smithi029:Deploy complete!
2022-03-13T04:22:38.392 INFO:teuthology.lock.ops:Checking smithi029.front.sepia.ceph.com
2022-03-13T04:22:38.393 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-13T04:22:38.415 INFO:teuthology.lock.ops:Checking smithi074.front.sepia.ceph.com
2022-03-13T04:22:38.415 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-13T04:22:38.437 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: set os type and version on server
2022-03-13T04:22:38.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2022-03-13T04:22:38.628 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64
2022-03-13T04:22:38.685 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream"
2022-03-13T04:22:38.685 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8"
2022-03-13T04:22:38.685 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos"
2022-03-13T04:22:38.685 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora"
2022-03-13T04:22:38.686 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8"
2022-03-13T04:22:38.686 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8"
2022-03-13T04:22:38.686 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-13T04:22:38.687 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31"
2022-03-13T04:22:38.687 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-13T04:22:38.687 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/"
2022-03-13T04:22:38.687 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-13T04:22:38.688 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-13T04:22:38.688 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-13T04:22:38.689 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2022-03-13T04:22:38.709 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: set os type and version on server
2022-03-13T04:22:38.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com'
2022-03-13T04:22:38.914 INFO:teuthology.orchestra.run.smithi074.stdout:x86_64
2022-03-13T04:22:38.974 INFO:teuthology.orchestra.run.smithi074.stdout:NAME="CentOS Stream"
2022-03-13T04:22:38.974 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION="8"
2022-03-13T04:22:38.974 INFO:teuthology.orchestra.run.smithi074.stdout:ID="centos"
2022-03-13T04:22:38.975 INFO:teuthology.orchestra.run.smithi074.stdout:ID_LIKE="rhel fedora"
2022-03-13T04:22:38.975 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION_ID="8"
2022-03-13T04:22:38.975 INFO:teuthology.orchestra.run.smithi074.stdout:PLATFORM_ID="platform:el8"
2022-03-13T04:22:38.975 INFO:teuthology.orchestra.run.smithi074.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-13T04:22:38.976 INFO:teuthology.orchestra.run.smithi074.stdout:ANSI_COLOR="0;31"
2022-03-13T04:22:38.976 INFO:teuthology.orchestra.run.smithi074.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-13T04:22:38.976 INFO:teuthology.orchestra.run.smithi074.stdout:HOME_URL="https://centos.org/"
2022-03-13T04:22:38.976 INFO:teuthology.orchestra.run.smithi074.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-13T04:22:38.976 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-13T04:22:38.977 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-13T04:22:38.977 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server
2022-03-13T04:22:39.072 INFO:teuthology.dispatcher.supervisor:Running job 6734042
2022-03-13T04:22:39.073 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6734042 --name teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6734042/orig.config.yaml
2022-03-13T04:22:39.076 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6734042
2022-03-13T04:22:39.077 INFO:teuthology.dispatcher.supervisor:Job PID: 14676
2022-03-13T04:22:39.077 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-13T10:53:08.828 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-13T10:53:08.963 INFO:teuthology.kill:Killing Pids: {14676}
2022-03-13T10:53:10.139 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2022-03-13T10:53:10.140 INFO:teuthology.task.internal:roles: ubuntu@smithi074.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2022-03-13T10:53:10.141 INFO:teuthology.misc:Compressing logs...
2022-03-13T10:53:10.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2022-03-13T10:53:10.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com'
2022-03-13T10:53:10.627 INFO:teuthology.misc:Compressing logs...
2022-03-13T10:53:11.218 INFO:teuthology.orchestra.run.smithi074.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-03-13T10:53:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-03-13T10:53:11.630 INFO:teuthology.misc:Compressing logs...
2022-03-13T10:53:11.660 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/ceph-client.admin.40078.log.gz already exists;	not overwritten
2022-03-13T10:53:15.889 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/teuthology/dispatcher/supervisor.py", line 287, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/teuthology/dispatcher/supervisor.py", line 345, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2022-03-13T10:53:15.951 INFO:teuthology.kill:No teuthology processes running
2022-03-13T10:53:17.095 INFO:teuthology.kill:Nuking machines: ['smithi029', 'smithi074']
2022-03-13T10:53:18.117 INFO:teuthology.kill:2022-03-13 10:53:18,117.117 INFO:teuthology.nuke:targets:
2022-03-13T10:53:18.118 INFO:teuthology.kill:  smithi029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDK/wtQrd7c3L8IP7a6hEoYISDClqTIdrbgEMZCHnXSaDJxRpUwa02N4ONOcExyYsmqn+Ugv97HYe829zMfMXLKhhZfwpiepMlSVN8/rmLDf1W4MuFbg1cD/wB6f8awY56nO45BwkvmG1C5DlEzlXCCPI0n387CGpydk1I1LLVMyxdxXZ6hWCPQSOkD3JWzh42ADwhrxbsPVLKiet0r2ehhGOdo8CKrJqML1BbXm0un7Sotdqyt2DgxwkyAB9mK9LVylOEOF1yjilLhFvLd1oRknZ1jxdqC5/wYZdffEMyqep3ulhEWwgSzHQnkjjEpNgjhMc6uwzUVKH0G55suCfr7Ifq+fDUi90+qsIKphkbZpN8zkJdn1ojoi1w9C1U+yf+nhDLOROPxUPae/niV5puI/+pb00nQchU1I1Fh/UKIdWpftD/bdFeA2JIrKOiM4/7LbOJPSPXoQFbtH4HcJ5xCj/J0aXL5apLXOr3ef+Xpch9Bi5c64mVNBgZcMWrGYVk=
2022-03-13T10:53:18.119 INFO:teuthology.kill:  smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIHPK6ImzcLR7BLv6jr5/y2CPcnlEYUgU7g0cMb3020UtLcoTkCFRMw2PFIz98u/xCzvdF28CBpY3BouAthl/l1bOXsxgmr9K2eqMMLPyM5CJNYcNxkTWrqISZoYomm/7Mz/T+Dp+ewo84PgjXFq36CRIj/aBzH7gjuLWBgZrC9KbuvTEASaJV+WuQWBngTj28OYm0DfGGbxlBJ85WUAlPFmdepejxBDsHxFgSDQFMyqMvTGWIWMV2IXJ7XtwVf7+O018dBq0aVELg/U1p+jY8KMeSwiRn9smapiScyd5rUko2iciilanlr+NGWN2F0wFWYkBd+J3EYutySHQ1LBuiCEdXdawCWo3A8WgA5Ji3ytIL/RRkfiXvU7bKXUoxf8AKky98jUi9TFCzISsbqWlzJ9e1umuB7CFE8CXx71Ytj6WSGLeZkzKrF97peV8NVkuSFpYFjMX0BKDEVMaFQP/mUlNWOcf/P3/Kp9Q8RASDWnS7I4fi6AzANT13bC9noHU=
2022-03-13T10:53:18.145 INFO:teuthology.kill:2022-03-13 10:53:18,144.144 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-13T10:53:18.149 INFO:teuthology.kill:2022-03-13 10:53:18,149.149 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-13T10:53:18.229 INFO:teuthology.kill:2022-03-13 10:53:18,229.229 INFO:teuthology.orchestra.console:Power off smithi029
2022-03-13T10:53:18.245 INFO:teuthology.kill:2022-03-13 10:53:18,245.245 INFO:teuthology.orchestra.console:Power off smithi074
2022-03-13T10:53:30.721 INFO:teuthology.kill:2022-03-13 10:53:30,721.721 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2022-03-13T10:53:30.729 INFO:teuthology.kill:2022-03-13 10:53:30,729.729 INFO:teuthology.orchestra.console:Power off for smithi074 completed
2022-03-13T10:53:30.877 INFO:teuthology.kill:2022-03-13 10:53:30,876.876 INFO:teuthology.lock.ops:unlocked: smithi029.front.sepia.ceph.com
2022-03-13T10:53:30.881 INFO:teuthology.kill:2022-03-13 10:53:30,881.881 INFO:teuthology.lock.ops:unlocked: smithi074.front.sepia.ceph.com
2022-03-13T10:55:31.205 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-13T10:55:31.312 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi029 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-13_07:01:03-rados-master-distro-default-smithi/6734733
2022-03-13T10:55:31.312 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi074 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-13_07:01:03-rados-master-distro-default-smithi/6734733