2021-11-19T03:54:52.665 INFO:root:teuthology version: 1.1.0-6cfddd5e
2021-11-19T03:54:52.798 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi025.front.sepia.ceph.com' reimaging
2021-11-19T03:54:52.799 INFO:teuthology.lock.ops:Updating [ubuntu@smithi025.front.sepia.ceph.com]: reset os type and version on server
2021-11-19T03:54:52.800 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server
2021-11-19T03:54:52.818 INFO:teuthology.lock.ops:Node 'ubuntu@smithi025.front.sepia.ceph.com' reimaging is complete
2021-11-19T03:54:52.819 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi052.front.sepia.ceph.com' reimaging
2021-11-19T03:54:52.819 INFO:teuthology.lock.ops:Updating [ubuntu@smithi052.front.sepia.ceph.com]: reset os type and version on server
2021-11-19T03:54:52.819 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server
2021-11-19T03:54:52.837 INFO:teuthology.lock.ops:Node 'ubuntu@smithi052.front.sepia.ceph.com' reimaging is complete
2021-11-19T03:54:53.041 INFO:teuthology.provision.fog.smithi025:Scheduling deploy of centos 8.3
2021-11-19T03:54:53.045 INFO:teuthology.provision.fog.smithi052:Scheduling deploy of centos 8.3
2021-11-19T03:54:53.386 INFO:teuthology.orchestra.console:Power off smithi052
2021-11-19T03:54:53.401 INFO:teuthology.orchestra.console:Power off smithi025
2021-11-19T03:55:01.757 INFO:teuthology.orchestra.console:Power off for smithi052 completed
2021-11-19T03:55:01.763 INFO:teuthology.orchestra.console:Power off for smithi025 completed
2021-11-19T03:55:01.858 INFO:teuthology.orchestra.console:Power on smithi052
2021-11-19T03:55:01.867 INFO:teuthology.orchestra.console:Power on smithi025
2021-11-19T03:55:10.235 INFO:teuthology.orchestra.console:Power on for smithi025 completed
2021-11-19T03:55:10.337 INFO:teuthology.provision.fog.smithi025:Waiting for deploy to finish
2021-11-19T03:55:10.349 INFO:teuthology.orchestra.console:Power on for smithi052 completed
2021-11-19T03:55:10.451 INFO:teuthology.provision.fog.smithi052:Waiting for deploy to finish
2021-11-19T03:59:44.786 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-19T03:59:45.076 INFO:teuthology.provision.fog.smithi052:Node is ready
2021-11-19T03:59:45.092 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052.front.sepia.ceph.com
2021-11-19T03:59:45.148 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52 smithi052.front.sepia.ceph.com smithi052
2021-11-19T03:59:45.462 INFO:teuthology.provision.fog.smithi052:Deploy complete!
2021-11-19T04:00:15.019 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-19T04:00:15.304 INFO:teuthology.provision.fog.smithi025:Node is ready
2021-11-19T04:00:15.320 INFO:teuthology.orchestra.run.smithi025.stdout:smithi025.front.sepia.ceph.com
2021-11-19T04:00:15.375 INFO:teuthology.orchestra.run.smithi025.stdout:172.21.15.25 smithi025.front.sepia.ceph.com smithi025
2021-11-19T04:00:15.716 INFO:teuthology.provision.fog.smithi025:Deploy complete!
2021-11-19T04:00:15.872 INFO:teuthology.lock.ops:Checking smithi025.front.sepia.ceph.com
2021-11-19T04:00:15.873 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-19T04:00:15.892 INFO:teuthology.lock.ops:Checking smithi052.front.sepia.ceph.com
2021-11-19T04:00:15.893 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-19T04:00:15.913 INFO:teuthology.lock.ops:Updating [smithi025.front.sepia.ceph.com]: set os type and version on server
2021-11-19T04:00:15.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi025.front.sepia.ceph.com'
2021-11-19T04:00:16.115 INFO:teuthology.orchestra.run.smithi025.stdout:x86_64
2021-11-19T04:00:16.169 INFO:teuthology.orchestra.run.smithi025.stdout:NAME="CentOS Linux"
2021-11-19T04:00:16.170 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION="8"
2021-11-19T04:00:16.170 INFO:teuthology.orchestra.run.smithi025.stdout:ID="centos"
2021-11-19T04:00:16.170 INFO:teuthology.orchestra.run.smithi025.stdout:ID_LIKE="rhel fedora"
2021-11-19T04:00:16.171 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION_ID="8"
2021-11-19T04:00:16.171 INFO:teuthology.orchestra.run.smithi025.stdout:PLATFORM_ID="platform:el8"
2021-11-19T04:00:16.171 INFO:teuthology.orchestra.run.smithi025.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-19T04:00:16.171 INFO:teuthology.orchestra.run.smithi025.stdout:ANSI_COLOR="0;31"
2021-11-19T04:00:16.172 INFO:teuthology.orchestra.run.smithi025.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-19T04:00:16.172 INFO:teuthology.orchestra.run.smithi025.stdout:HOME_URL="https://centos.org/"
2021-11-19T04:00:16.172 INFO:teuthology.orchestra.run.smithi025.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-19T04:00:16.172 INFO:teuthology.orchestra.run.smithi025.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-19T04:00:16.172 INFO:teuthology.orchestra.run.smithi025.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-19T04:00:16.174 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server
2021-11-19T04:00:16.192 INFO:teuthology.lock.ops:Updating [smithi052.front.sepia.ceph.com]: set os type and version on server
2021-11-19T04:00:16.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2021-11-19T04:00:16.466 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64
2021-11-19T04:00:16.522 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Linux"
2021-11-19T04:00:16.522 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="8"
2021-11-19T04:00:16.522 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos"
2021-11-19T04:00:16.523 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora"
2021-11-19T04:00:16.523 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="8"
2021-11-19T04:00:16.523 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el8"
2021-11-19T04:00:16.523 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-19T04:00:16.524 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31"
2021-11-19T04:00:16.524 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-19T04:00:16.524 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/"
2021-11-19T04:00:16.524 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-19T04:00:16.524 INFO:teuthology.orchestra.run.smithi052.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-19T04:00:16.525 INFO:teuthology.orchestra.run.smithi052.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-19T04:00:16.525 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server
2021-11-19T04:00:16.621 INFO:teuthology.dispatcher.supervisor:Running job 6505968
2021-11-19T04:00:16.622 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-11-16_04:17:03-fs-pacific-distro-default-smithi/6505968 --name teuthology-2021-11-16_04:17:03-fs-pacific-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/teuthology-2021-11-16_04:17:03-fs-pacific-distro-default-smithi/6505968/orig.config.yaml
2021-11-19T04:00:16.625 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-11-16_04:17:03-fs-pacific-distro-default-smithi/6505968
2021-11-19T04:00:16.626 INFO:teuthology.dispatcher.supervisor:Job PID: 12562
2021-11-19T04:00:16.626 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-19T16:01:11.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-19T16:01:11.491 INFO:teuthology.kill:Killing Pids: {12562}
2021-11-19T16:01:12.754 INFO:teuthology.task.internal:roles: ubuntu@smithi025.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-19T16:01:12.755 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-19T16:01:12.755 INFO:teuthology.misc:Compressing logs...
2021-11-19T16:01:12.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi025.front.sepia.ceph.com'
2021-11-19T16:01:12.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2021-11-19T16:01:13.474 INFO:teuthology.misc:Compressing logs...
2021-11-19T16:01:13.510 INFO:teuthology.orchestra.run.smithi025.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-19T16:01:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-19T16:01:13.579 INFO:teuthology.orchestra.run.smithi025.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-19T16:01:13.580 INFO:teuthology.orchestra.run.smithi025.stderr:tar: Error is not recoverable: exiting now
2021-11-19T16:01:13.611 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-19T16:01:13.611 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2021-11-19T16:01:13.612 INFO:teuthology.misc:Compressing logs...
2021-11-19T16:01:13.654 INFO:teuthology.orchestra.run.smithi025.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-19T16:01:13.657 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi025 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-19T16:01:13.726 INFO:teuthology.kill:No teuthology processes running
2021-11-19T16:01:14.873 INFO:teuthology.kill:Nuking machines: ['smithi025', 'smithi052']
2021-11-19T16:01:15.875 INFO:teuthology.kill:2021-11-19 16:01:15,875.875 INFO:teuthology.nuke:targets:
2021-11-19T16:01:15.876 INFO:teuthology.kill:  smithi025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0iDq3ckphpqiHqHQmHGpJijJP11cget2zlw1aFg8xBD6AWyY+TgDBoUygiI5IxVSopbK3zD1h+5NB6w2nfe/teUJv5pvZpsQ5WVjUFBsZ1e8Suu8wc57Ov4L3Xa27Q4gyDIqETNDz/Arl+vIHOtIsuabgvqOaTKwtmHLTbgS1+17GSpAUTtbbGwi9z1ic1FrGLvIt/d0+uhKs+ss1u+Rbg1+2ee+J5+YvFZJvhXqPXQel3agHa1RUwuW6DyzThXzqVnP+gkHkKidjKz068QBnZnL6RuW/orVA+qKoHaJNrmpMfe2oCayHqcIAwM2JTXf2vnMpViehNM6lalZpw+LuLm3HmUfc3vNuSpapKpY0asv1itmtGHMBudKscI4WvgnX/VlXjlhsE4hzroNnd3tNYYUI56aHAGHr0GhyCVqVAOZeKzeLdtHWYkhhyXnvSgyb0uiZMNDLKVwSsIB6zpuvXOfhn5P/HFsiQ0qiRYyW6aSOhCBzETMDvM2BgoN2UtU=
2021-11-19T16:01:15.878 INFO:teuthology.kill:  smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEIwQIAT0p7pepgpws6UMpZozYLvzkhHp3+7uvJYAdA6JEBY/XuihDE7txbOprbCQNe3+HFB/FZwvdUDlQ9di3heKBiFuQ7/2eca+THSKsTKql56x5M5qPoIg9eEDutsD3+VoIWZ2zMOiH+pfDbeD0pl4eeGumIMzkf+6Gir8cgL1J6YgSNFQkEisyyvjJBuusc+R120M5tkbfNCu50Fhh50FEFdqjOnHkY5WZ5n01r6zVx9XoTvKhdhm3Wf8mlw6SB1pYVEcoGje6k/YDwvk4jTdDnpjS4H0rvJ4s5v1tlNPD5VYuhUae0uT0ZSNMnsn4TcHIfFF4oCgtuvT1EgVeAzQ5CZgC/IITBVC4a1gTOUxS1iouuy/Zf+9AyiBX3wF+j+KWUY4kmP+6hsa4DXYJx0nf10kihKP2s5M2toDUhRURQO9bgHYhu10+RHUYl8flZFqBT2SrVbhlY3wcLb2bCapQ0jdSUVUerHBQxlMEABoOPHH+TBCAUx7qmhEDwfU=
2021-11-19T16:01:15.903 INFO:teuthology.kill:2021-11-19 16:01:15,903.903 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-19T16:01:15.907 INFO:teuthology.kill:2021-11-19 16:01:15,907.907 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-19T16:01:15.992 INFO:teuthology.kill:2021-11-19 16:01:15,991.991 INFO:teuthology.orchestra.console:Power off smithi052
2021-11-19T16:01:16.007 INFO:teuthology.kill:2021-11-19 16:01:16,007.007 INFO:teuthology.orchestra.console:Power off smithi025
2021-11-19T16:01:24.366 INFO:teuthology.kill:2021-11-19 16:01:24,366.366 INFO:teuthology.orchestra.console:Power off for smithi052 completed
2021-11-19T16:01:24.375 INFO:teuthology.kill:2021-11-19 16:01:24,374.374 INFO:teuthology.orchestra.console:Power off for smithi025 completed
2021-11-19T16:01:24.513 INFO:teuthology.kill:2021-11-19 16:01:24,512.512 INFO:teuthology.lock.ops:unlocked: smithi052.front.sepia.ceph.com
2021-11-19T16:01:24.515 INFO:teuthology.kill:2021-11-19 16:01:24,515.515 INFO:teuthology.lock.ops:unlocked: smithi025.front.sepia.ceph.com
2021-11-19T16:03:24.850 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-19T16:03:25.008 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi025 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-19_15:58:40-rbd-wip-yuri5-testing-2021-11-19-0638-pacific-distro-basic-smithi/6513940
2021-11-19T16:03:25.009 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi052 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-19_15:58:40-rbd-wip-yuri5-testing-2021-11-19-0638-pacific-distro-basic-smithi/6513941