2023-01-14T02:41:59.483 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2023-01-14T02:41:59.600 INFO:teuthology.lock.ops:Start node 'smithi132.front.sepia.ceph.com' reimaging
2023-01-14T02:41:59.601 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: reset os type and version on server
2023-01-14T02:41:59.602 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server
2023-01-14T02:41:59.620 INFO:teuthology.lock.ops:Node 'smithi132.front.sepia.ceph.com' reimaging is complete
2023-01-14T02:41:59.620 INFO:teuthology.lock.ops:Start node 'smithi145.front.sepia.ceph.com' reimaging
2023-01-14T02:41:59.621 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: reset os type and version on server
2023-01-14T02:41:59.621 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server
2023-01-14T02:41:59.639 INFO:teuthology.lock.ops:Node 'smithi145.front.sepia.ceph.com' reimaging is complete
2023-01-14T02:41:59.774 INFO:teuthology.provision.fog.smithi132:Scheduling deploy of centos 8.stream
2023-01-14T02:41:59.780 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of centos 8.stream
2023-01-14T02:41:59.971 INFO:teuthology.orchestra.console:Power off smithi145
2023-01-14T02:41:59.987 INFO:teuthology.orchestra.console:Power off smithi132
2023-01-14T02:42:00.119 INFO:teuthology.orchestra.console:Power off for smithi145 completed
2023-01-14T02:42:00.131 INFO:teuthology.orchestra.console:Power off for smithi132 completed
2023-01-14T02:42:00.221 INFO:teuthology.orchestra.console:Power on smithi145
2023-01-14T02:42:00.232 INFO:teuthology.orchestra.console:Power on smithi132
2023-01-14T02:42:04.485 INFO:teuthology.orchestra.console:Power on for smithi132 completed
2023-01-14T02:42:04.587 INFO:teuthology.provision.fog.smithi132:Waiting for deploy to finish
2023-01-14T02:42:08.590 INFO:teuthology.orchestra.console:Power on for smithi145 completed
2023-01-14T02:42:08.691 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish
2023-01-14T02:44:53.584 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-01-14T02:45:02.668 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-01-14T02:45:11.724 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-01-14T02:45:12.784 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145
2023-01-14T02:45:20.780 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-01-14T02:45:21.836 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145
2023-01-14T02:45:29.840 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-01-14T02:45:30.896 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145
2023-01-14T02:45:39.948 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145
2023-01-14T02:46:07.576 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-14T02:46:17.865 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-14T02:46:32.665 INFO:teuthology.provision.fog.smithi132:Node is ready
2023-01-14T02:46:32.687 INFO:teuthology.orchestra.run.smithi132.stdout:smithi132.front.sepia.ceph.com
2023-01-14T02:46:32.747 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132 smithi132.front.sepia.ceph.com smithi132
2023-01-14T02:46:33.117 INFO:teuthology.provision.fog.smithi132:Deploy complete!
2023-01-14T02:46:37.972 INFO:teuthology.provision.fog.smithi145:Node is ready
2023-01-14T02:46:37.993 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145.front.sepia.ceph.com
2023-01-14T02:46:38.052 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145 smithi145.front.sepia.ceph.com smithi145
2023-01-14T02:46:38.451 INFO:teuthology.provision.fog.smithi145:Deploy complete!
2023-01-14T02:46:38.617 INFO:teuthology.lock.ops:Checking smithi132.front.sepia.ceph.com
2023-01-14T02:46:38.618 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-14T02:46:38.638 INFO:teuthology.lock.ops:Checking smithi145.front.sepia.ceph.com
2023-01-14T02:46:38.639 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-14T02:46:38.655 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: set os type and version on server
2023-01-14T02:46:38.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com'
2023-01-14T02:46:38.853 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64
2023-01-14T02:46:38.911 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream"
2023-01-14T02:46:38.912 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="8"
2023-01-14T02:46:38.912 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos"
2023-01-14T02:46:38.912 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora"
2023-01-14T02:46:38.912 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="8"
2023-01-14T02:46:38.913 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el8"
2023-01-14T02:46:38.913 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 8"
2023-01-14T02:46:38.913 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31"
2023-01-14T02:46:38.913 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-01-14T02:46:38.913 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/"
2023-01-14T02:46:38.914 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-14T02:46:38.914 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-14T02:46:38.914 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-01-14T02:46:38.915 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server
2023-01-14T02:46:38.931 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: set os type and version on server
2023-01-14T02:46:38.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com'
2023-01-14T02:46:39.078 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64
2023-01-14T02:46:39.135 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Stream"
2023-01-14T02:46:39.136 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="8"
2023-01-14T02:46:39.136 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos"
2023-01-14T02:46:39.136 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora"
2023-01-14T02:46:39.136 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="8"
2023-01-14T02:46:39.136 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el8"
2023-01-14T02:46:39.137 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Stream 8"
2023-01-14T02:46:39.137 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31"
2023-01-14T02:46:39.137 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-01-14T02:46:39.137 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://centos.org/"
2023-01-14T02:46:39.138 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-14T02:46:39.138 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-14T02:46:39.138 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-01-14T02:46:39.138 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server
2023-01-14T02:46:39.243 INFO:teuthology.dispatcher.supervisor:Running job 7141733
2023-01-14T02:46:39.243 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-13_17:02:58-fs-wip-yuri5-testing-2023-01-12-1540-quincy-distro-default-smithi/7141733 --name yuriw-2023-01-13_17:02:58-fs-wip-yuri5-testing-2023-01-12-1540-quincy-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-01-13_17:02:58-fs-wip-yuri5-testing-2023-01-12-1540-quincy-distro-default-smithi/7141733/orig.config.yaml
2023-01-14T02:46:39.248 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-01-13_17:02:58-fs-wip-yuri5-testing-2023-01-12-1540-quincy-distro-default-smithi/7141733
2023-01-14T02:46:39.248 INFO:teuthology.dispatcher.supervisor:Job PID: 4036
2023-01-14T02:46:39.249 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-01-14T14:48:09.060 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:48:09.238 INFO:teuthology.kill:Killing Pids: {4036}
2023-01-14T14:48:10.040 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2023-01-14T14:48:10.041 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2023-01-14T14:48:10.041 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:48:10.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com'
2023-01-14T14:48:10.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com'
2023-01-14T14:48:10.470 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:48:10.679 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:48:10.708 INFO:teuthology.orchestra.run.smithi132.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-01-14T14:48:10.717 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:48:10.783 INFO:teuthology.kill:No teuthology processes running
2023-01-14T14:48:11.526 INFO:teuthology.kill:Nuking machines: ['smithi132', 'smithi145']
2023-01-14T14:48:12.134 INFO:teuthology.kill:2023-01-14 14:48:12,133.133 INFO:teuthology.nuke:targets:
2023-01-14T14:48:12.134 INFO:teuthology.kill:  smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKwJq4JYB/nmQIwi5Q7Z4X1R7E0nsANommxyFI0LeVNm6sLBk7A0MrfkTD72pNKcMsWuk1SdMxmbLM1dxv8F1Uk=
2023-01-14T14:48:12.135 INFO:teuthology.kill:  smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxigEDIhBpezWxMzsXQQsCHAFDYIvJFqWgpdvlFiz9UeUybVyfiYGf82M8MKPcLXlZQi/h7Dx+pFJKCkbdO2B6mp1RQiUQ1/NVdnZgMvkn8Dhem+Um5bNDcsNSe6S7XBZJlkWliRub0+EmzyfieVRzgWHz8ZPMOMAxWULhYzux9pjyhudrY8HLZiI5WZDlaPZjX634z9yfAXp7bvcxbMXAxRojQYwRzNPE8gb3gzh948qPdI8x7SKQcYcO3/zQ3yMzPcipRN/KVnxX3jwim5xZw1RFpnp3kBrqrYuAEGOoq82py/6tJJNEyL95S9IAt9CCFTBegWuBF+OAt24nO38B0d0070QobfpF4Y9ftaJ0d5bBWWaLVly+mY8BBzUAuEvSqjy7s1P+oQb//iGUeeYaJTGpz+TnztHiAcpndoplNwQUwf4AM0RBmsM/VOElMeuQ11OOHxCRKZdG5ePKJ0OKaUlSe1E+I7JVyR3HEfI0w0YK2hRCFUo6poSXOo0zack=
2023-01-14T14:48:12.135 INFO:teuthology.kill:2023-01-14 14:48:12,134.134 INFO:teuthology.nuke:Checking targets against current locks
2023-01-14T14:48:12.199 INFO:teuthology.kill:2023-01-14 14:48:12,199.199 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-14T14:48:12.225 INFO:teuthology.kill:2023-01-14 14:48:12,224.224 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-14T14:48:12.285 INFO:teuthology.kill:2023-01-14 14:48:12,285.285 INFO:teuthology.orchestra.console:Power off smithi132
2023-01-14T14:48:12.322 INFO:teuthology.kill:2023-01-14 14:48:12,321.321 INFO:teuthology.orchestra.console:Power off smithi145
2023-01-14T14:48:20.662 INFO:teuthology.kill:2023-01-14 14:48:20,661.661 INFO:teuthology.orchestra.console:Power off for smithi132 completed
2023-01-14T14:48:20.691 INFO:teuthology.kill:2023-01-14 14:48:20,691.691 INFO:teuthology.orchestra.console:Power off for smithi145 completed
2023-01-14T14:48:20.804 INFO:teuthology.kill:2023-01-14 14:48:20,803.803 INFO:teuthology.lock.ops:unlocked: smithi132.front.sepia.ceph.com
2023-01-14T14:48:20.831 INFO:teuthology.kill:2023-01-14 14:48:20,831.831 INFO:teuthology.lock.ops:unlocked: smithi145.front.sepia.ceph.com
2023-01-14T14:50:21.116 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-01-14T14:50:21.216 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi132 but it was locked by another job: /home/teuthworker/archive/aemerson-2023-01-13_23:20:40-rgw-wip-rgw-multisite-reshard-reef-distro-default-smithi/7142117
2023-01-14T14:50:21.216 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi145 but it was locked by another job: /home/teuthworker/archive/aemerson-2023-01-13_23:20:40-rgw-wip-rgw-multisite-reshard-reef-distro-default-smithi/7142117