2021-11-24T02:14:11.250 INFO:root:teuthology version: 1.1.0-6cfddd5e
2021-11-24T02:14:11.377 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi152.front.sepia.ceph.com' reimaging
2021-11-24T02:14:11.380 INFO:teuthology.lock.ops:Updating [ubuntu@smithi152.front.sepia.ceph.com]: reset os type and version on server
2021-11-24T02:14:11.380 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server
2021-11-24T02:14:11.400 INFO:teuthology.lock.ops:Node 'ubuntu@smithi152.front.sepia.ceph.com' reimaging is complete
2021-11-24T02:14:11.401 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi173.front.sepia.ceph.com' reimaging
2021-11-24T02:14:11.401 INFO:teuthology.lock.ops:Updating [ubuntu@smithi173.front.sepia.ceph.com]: reset os type and version on server
2021-11-24T02:14:11.401 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server
2021-11-24T02:14:11.423 INFO:teuthology.lock.ops:Node 'ubuntu@smithi173.front.sepia.ceph.com' reimaging is complete
2021-11-24T02:14:11.614 INFO:teuthology.provision.fog.smithi152:Scheduling deploy of centos 8.3
2021-11-24T02:14:11.632 INFO:teuthology.provision.fog.smithi173:Scheduling deploy of centos 8.3
2021-11-24T02:14:11.973 INFO:teuthology.orchestra.console:Power off smithi152
2021-11-24T02:14:11.993 INFO:teuthology.orchestra.console:Power off smithi173
2021-11-24T02:14:24.460 INFO:teuthology.orchestra.console:Power off for smithi152 completed
2021-11-24T02:14:24.467 INFO:teuthology.orchestra.console:Power off for smithi173 completed
2021-11-24T02:14:24.562 INFO:teuthology.orchestra.console:Power on smithi152
2021-11-24T02:14:24.569 INFO:teuthology.orchestra.console:Power on smithi173
2021-11-24T02:14:32.928 INFO:teuthology.orchestra.console:Power on for smithi152 completed
2021-11-24T02:14:32.934 INFO:teuthology.orchestra.console:Power on for smithi173 completed
2021-11-24T02:14:33.029 INFO:teuthology.provision.fog.smithi152:Waiting for deploy to finish
2021-11-24T02:14:33.035 INFO:teuthology.provision.fog.smithi173:Waiting for deploy to finish
2021-11-24T02:19:22.656 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-24T02:19:22.964 INFO:teuthology.provision.fog.smithi173:Node is ready
2021-11-24T02:19:22.980 INFO:teuthology.orchestra.run.smithi173.stdout:smithi173.front.sepia.ceph.com
2021-11-24T02:19:23.035 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.15.173 smithi173.front.sepia.ceph.com smithi173
2021-11-24T02:19:23.358 INFO:teuthology.provision.fog.smithi173:Deploy complete!
2021-11-24T02:19:37.883 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-24T02:19:38.181 INFO:teuthology.provision.fog.smithi152:Node is ready
2021-11-24T02:19:38.196 INFO:teuthology.orchestra.run.smithi152.stdout:smithi152.front.sepia.ceph.com
2021-11-24T02:19:38.252 INFO:teuthology.orchestra.run.smithi152.stdout:172.21.15.152 smithi152.front.sepia.ceph.com smithi152
2021-11-24T02:19:38.577 INFO:teuthology.provision.fog.smithi152:Deploy complete!
2021-11-24T02:19:38.735 INFO:teuthology.lock.ops:Checking smithi152.front.sepia.ceph.com
2021-11-24T02:19:38.736 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-24T02:19:38.757 INFO:teuthology.lock.ops:Checking smithi173.front.sepia.ceph.com
2021-11-24T02:19:38.757 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-24T02:19:38.778 INFO:teuthology.lock.ops:Updating [smithi152.front.sepia.ceph.com]: set os type and version on server
2021-11-24T02:19:38.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com'
2021-11-24T02:19:38.980 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64
2021-11-24T02:19:39.034 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Linux"
2021-11-24T02:19:39.034 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="8"
2021-11-24T02:19:39.034 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos"
2021-11-24T02:19:39.035 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora"
2021-11-24T02:19:39.035 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="8"
2021-11-24T02:19:39.035 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el8"
2021-11-24T02:19:39.036 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-24T02:19:39.036 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31"
2021-11-24T02:19:39.036 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-24T02:19:39.036 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/"
2021-11-24T02:19:39.037 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-24T02:19:39.037 INFO:teuthology.orchestra.run.smithi152.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-24T02:19:39.037 INFO:teuthology.orchestra.run.smithi152.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-24T02:19:39.038 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server
2021-11-24T02:19:39.056 INFO:teuthology.lock.ops:Updating [smithi173.front.sepia.ceph.com]: set os type and version on server
2021-11-24T02:19:39.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com'
2021-11-24T02:19:39.324 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64
2021-11-24T02:19:39.378 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Linux"
2021-11-24T02:19:39.379 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="8"
2021-11-24T02:19:39.379 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos"
2021-11-24T02:19:39.379 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora"
2021-11-24T02:19:39.379 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="8"
2021-11-24T02:19:39.380 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el8"
2021-11-24T02:19:39.380 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-24T02:19:39.380 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31"
2021-11-24T02:19:39.380 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-24T02:19:39.381 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/"
2021-11-24T02:19:39.381 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-24T02:19:39.381 INFO:teuthology.orchestra.run.smithi173.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-24T02:19:39.381 INFO:teuthology.orchestra.run.smithi173.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-24T02:19:39.382 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server
2021-11-24T02:19:39.489 INFO:teuthology.dispatcher.supervisor:Running job 6522103
2021-11-24T02:19:39.489 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_ifed01@teuthology --archive /home/teuthworker/archive/ifed01-2021-11-23_11:51:25-rados-wip-ifed-better-daemonperf-distro-default-smithi/6522103 --name ifed01-2021-11-23_11:51:25-rados-wip-ifed-better-daemonperf-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/ifed01-2021-11-23_11:51:25-rados-wip-ifed-better-daemonperf-distro-default-smithi/6522103/orig.config.yaml
2021-11-24T02:19:39.493 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ifed01-2021-11-23_11:51:25-rados-wip-ifed-better-daemonperf-distro-default-smithi/6522103
2021-11-24T02:19:39.493 INFO:teuthology.dispatcher.supervisor:Job PID: 5399
2021-11-24T02:19:39.494 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-24T14:20:35.893 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-24T14:20:36.106 INFO:teuthology.kill:Killing Pids: {5399}
2021-11-24T14:20:37.345 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-24T14:20:37.346 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-24T14:20:37.347 INFO:teuthology.misc:Compressing logs...
2021-11-24T14:20:37.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com'
2021-11-24T14:20:37.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com'
2021-11-24T14:20:37.931 INFO:teuthology.misc:Compressing logs...
2021-11-24T14:20:37.970 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-24T14:20:37.995 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-24T14:20:38.036 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-24T14:20:38.036 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now
2021-11-24T14:20:38.060 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-24T14:20:38.060 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now
2021-11-24T14:20:38.062 INFO:teuthology.misc:Compressing logs...
2021-11-24T14:20:38.104 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-24T14:20:38.105 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi152 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-24T14:20:38.174 INFO:teuthology.kill:No teuthology processes running
2021-11-24T14:20:39.316 INFO:teuthology.kill:Nuking machines: ['smithi152', 'smithi173']
2021-11-24T14:20:40.296 INFO:teuthology.kill:2021-11-24 14:20:40,295.295 INFO:teuthology.nuke:targets:
2021-11-24T14:20:40.297 INFO:teuthology.kill:  smithi152.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxdjoWoKqq4oWZnN2LhF4aVnV102zpdEjrLHY/vvSGIaB6OHOc7nZxv+XRwxudF3NbZF0M2gF+cHYHd+fckKS1BvMqV+XeBS4/aad83hhU8DA5F9bEQNWr/A318t7iYe3MVPJYEBLnwNRR0PL2812H8FAWVRshaKdfuzYrCFVQKdRPXdtNWPPPpaOIAgH8W1KD3g1XSMnwhAHlV/xki5L2VfZoCqZbgn8/kVaWx6AMhCoJ7KnSvFxrZJIO1SFuDOuJWofYkfG405ZyTKT1fuxzfwa2kkpFvTr+aIgbrpNuj5JViOpUVBf8oZF89hGZojskrn2NRB/F7WPjPZzACf8yUpSOJBIi4/s56ncRVh0YfImW16zsmvm0yo8mZSjPitOXugKrY6HLOdSGWCA4s7RxWAqaOQ8m9yTe0m2YPBVKk2SQcM1UVysyR9k3x/EQW6QU1tTWPKDCXwM2YCT+fU9tcuLJjkT7L3Il7/RvHLijRxuOe2V8zuWd7AmSfbv4D/M=
2021-11-24T14:20:40.297 INFO:teuthology.kill:  smithi173.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuPGa9f0sAwJeMytT9GXhC4s8pZz4V38JEswGkyoPRTofmKNtUZ6Zf/fD3CO+MUo2hWjfk8fP1zquEySU6HZ8CDfu4W4KDPtnAxuyIXrYCDp+QL6fHvXCaUabaUDdFs2oYkWnmKboufuylhURAz2ljnMMtsNmlubk81ykknnjbfbcmYFG6JeyGiCSr1AI1SNyp5bGi6GMuRXrVxmA7L5F73E3r7P+bfBnE9seJhNHl1ofjqEWqEeLVqJ9o4t0AvgEcCyBiiH1ExB+4Z+o03Bl62WveZV+8wHeT64YiAAsI7IXBainnv6Fr0ExRffmh23nE5N0MOgIH8q6t6q8CWl4pOR82jzVG9C/NJId53dTtMYHYGFOdfnF3pfxbUGUTq18FvMTUUAHXQAs9s5OrOLCrY7OpR7gnJCW6snZ1Md8zIjfwyxBSpOeK9i85SgyVrV2Y4rPyFBTPQIbtcAOCjQRsAqwoR9U9ALGy9PKHTjoq8j+b4rAgrtZRAF1rZ/vDrIU=
2021-11-24T14:20:40.323 INFO:teuthology.kill:2021-11-24 14:20:40,323.323 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-24T14:20:40.329 INFO:teuthology.kill:2021-11-24 14:20:40,329.329 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-24T14:20:40.408 INFO:teuthology.kill:2021-11-24 14:20:40,408.408 INFO:teuthology.orchestra.console:Power off smithi173
2021-11-24T14:20:40.421 INFO:teuthology.kill:2021-11-24 14:20:40,421.421 INFO:teuthology.orchestra.console:Power off smithi152
2021-11-24T14:20:48.780 INFO:teuthology.kill:2021-11-24 14:20:48,779.779 INFO:teuthology.orchestra.console:Power off for smithi173 completed
2021-11-24T14:20:48.784 INFO:teuthology.kill:2021-11-24 14:20:48,784.784 INFO:teuthology.orchestra.console:Power off for smithi152 completed
2021-11-24T14:20:48.929 INFO:teuthology.kill:2021-11-24 14:20:48,929.929 INFO:teuthology.lock.ops:unlocked: smithi173.front.sepia.ceph.com
2021-11-24T14:20:48.933 INFO:teuthology.kill:2021-11-24 14:20:48,933.933 INFO:teuthology.lock.ops:unlocked: smithi152.front.sepia.ceph.com
2021-11-24T14:22:49.244 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-24T14:22:49.360 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi152 but it was locked by another job: /home/teuthworker/archive/gsalomon-2021-11-24_13:58:57-rgw:verify-wip-test-albin-error-status-distro-basic-smithi/6524311
2021-11-24T14:22:49.360 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi173 but it was locked by another job: /home/teuthworker/archive/gsalomon-2021-11-24_13:58:57-rgw:verify-wip-test-albin-error-status-distro-basic-smithi/6524311