2021-11-18T23:37:10.375 INFO:root:teuthology version: 1.1.0-6cfddd5e
2021-11-18T23:37:10.508 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi079.front.sepia.ceph.com' reimaging
2021-11-18T23:37:10.509 INFO:teuthology.lock.ops:Updating [ubuntu@smithi079.front.sepia.ceph.com]: reset os type and version on server
2021-11-18T23:37:10.509 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server
2021-11-18T23:37:10.580 INFO:teuthology.lock.ops:Node 'ubuntu@smithi079.front.sepia.ceph.com' reimaging is complete
2021-11-18T23:37:10.581 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi194.front.sepia.ceph.com' reimaging
2021-11-18T23:37:10.581 INFO:teuthology.lock.ops:Updating [ubuntu@smithi194.front.sepia.ceph.com]: reset os type and version on server
2021-11-18T23:37:10.582 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server
2021-11-18T23:37:10.610 INFO:teuthology.lock.ops:Node 'ubuntu@smithi194.front.sepia.ceph.com' reimaging is complete
2021-11-18T23:37:10.802 INFO:teuthology.provision.fog.smithi079:Scheduling deploy of centos 8.3
2021-11-18T23:37:10.827 INFO:teuthology.provision.fog.smithi194:Scheduling deploy of centos 8.3
2021-11-18T23:37:11.150 INFO:teuthology.orchestra.console:Power off smithi079
2021-11-18T23:37:11.171 INFO:teuthology.orchestra.console:Power off smithi194
2021-11-18T23:37:11.305 INFO:teuthology.orchestra.console:Power off for smithi079 completed
2021-11-18T23:37:11.318 INFO:teuthology.orchestra.console:Power off for smithi194 completed
2021-11-18T23:37:11.406 INFO:teuthology.orchestra.console:Power on smithi079
2021-11-18T23:37:11.420 INFO:teuthology.orchestra.console:Power on smithi194
2021-11-18T23:37:19.794 INFO:teuthology.orchestra.console:Power on for smithi079 completed
2021-11-18T23:37:19.803 INFO:teuthology.orchestra.console:Power on for smithi194 completed
2021-11-18T23:37:19.896 INFO:teuthology.provision.fog.smithi079:Waiting for deploy to finish
2021-11-18T23:37:19.904 INFO:teuthology.provision.fog.smithi194:Waiting for deploy to finish
2021-11-18T23:42:09.451 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-18T23:42:09.484 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-18T23:42:09.748 INFO:teuthology.provision.fog.smithi194:Node is ready
2021-11-18T23:42:09.804 INFO:teuthology.orchestra.run.smithi194.stdout:smithi194.front.sepia.ceph.com
2021-11-18T23:42:09.807 INFO:teuthology.provision.fog.smithi079:Node is ready
2021-11-18T23:42:09.822 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079.front.sepia.ceph.com
2021-11-18T23:42:09.860 INFO:teuthology.orchestra.run.smithi194.stdout:172.21.15.194 smithi194.front.sepia.ceph.com smithi194
2021-11-18T23:42:09.878 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79 smithi079.front.sepia.ceph.com smithi079
2021-11-18T23:42:10.223 INFO:teuthology.provision.fog.smithi194:Deploy complete!
2021-11-18T23:42:10.301 INFO:teuthology.provision.fog.smithi079:Deploy complete!
2021-11-18T23:42:10.556 INFO:teuthology.lock.ops:Checking smithi079.front.sepia.ceph.com
2021-11-18T23:42:10.557 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-18T23:42:10.582 INFO:teuthology.lock.ops:Checking smithi194.front.sepia.ceph.com
2021-11-18T23:42:10.582 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-18T23:42:10.607 INFO:teuthology.lock.ops:Updating [smithi079.front.sepia.ceph.com]: set os type and version on server
2021-11-18T23:42:10.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com'
2021-11-18T23:42:10.915 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64
2021-11-18T23:42:10.970 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Linux"
2021-11-18T23:42:10.970 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="8"
2021-11-18T23:42:10.970 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos"
2021-11-18T23:42:10.971 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora"
2021-11-18T23:42:10.971 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="8"
2021-11-18T23:42:10.971 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el8"
2021-11-18T23:42:10.972 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-18T23:42:10.972 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31"
2021-11-18T23:42:10.972 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-18T23:42:10.972 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/"
2021-11-18T23:42:10.973 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-18T23:42:10.973 INFO:teuthology.orchestra.run.smithi079.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-18T23:42:10.973 INFO:teuthology.orchestra.run.smithi079.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-18T23:42:10.974 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server
2021-11-18T23:42:10.994 INFO:teuthology.lock.ops:Updating [smithi194.front.sepia.ceph.com]: set os type and version on server
2021-11-18T23:42:10.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com'
2021-11-18T23:42:11.270 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64
2021-11-18T23:42:11.325 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Linux"
2021-11-18T23:42:11.325 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="8"
2021-11-18T23:42:11.326 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos"
2021-11-18T23:42:11.326 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora"
2021-11-18T23:42:11.326 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="8"
2021-11-18T23:42:11.327 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el8"
2021-11-18T23:42:11.327 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-18T23:42:11.327 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31"
2021-11-18T23:42:11.327 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-18T23:42:11.328 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/"
2021-11-18T23:42:11.328 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-18T23:42:11.328 INFO:teuthology.orchestra.run.smithi194.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-18T23:42:11.328 INFO:teuthology.orchestra.run.smithi194.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-18T23:42:11.329 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server
2021-11-18T23:42:11.436 INFO:teuthology.dispatcher.supervisor:Running job 6509252
2021-11-18T23:42:11.437 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_ifed01@teuthology --archive /home/teuthworker/archive/ifed01-2021-11-17_10:00:50-rados-wip-ifed-better-daemonperf-distro-default-smithi/6509252 --name ifed01-2021-11-17_10:00:50-rados-wip-ifed-better-daemonperf-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/ifed01-2021-11-17_10:00:50-rados-wip-ifed-better-daemonperf-distro-default-smithi/6509252/orig.config.yaml
2021-11-18T23:42:11.442 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ifed01-2021-11-17_10:00:50-rados-wip-ifed-better-daemonperf-distro-default-smithi/6509252
2021-11-18T23:42:11.443 INFO:teuthology.dispatcher.supervisor:Job PID: 12996
2021-11-18T23:42:11.443 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-19T11:43:07.737 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-19T11:43:07.946 INFO:teuthology.kill:Killing Pids: {12996}
2021-11-19T11:43:09.167 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-19T11:43:09.168 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-19T11:43:09.168 INFO:teuthology.misc:Compressing logs...
2021-11-19T11:43:09.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com'
2021-11-19T11:43:09.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com'
2021-11-19T11:43:09.757 INFO:teuthology.misc:Compressing logs...
2021-11-19T11:43:09.796 INFO:teuthology.orchestra.run.smithi079.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-19T11:43:09.820 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-19T11:43:09.864 INFO:teuthology.orchestra.run.smithi079.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-19T11:43:09.864 INFO:teuthology.orchestra.run.smithi079.stderr:tar: Error is not recoverable: exiting now
2021-11-19T11:43:09.888 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-19T11:43:09.888 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now
2021-11-19T11:43:09.889 INFO:teuthology.misc:Compressing logs...
2021-11-19T11:43:09.932 INFO:teuthology.orchestra.run.smithi079.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-19T11:43:09.933 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-19T11:43:10.001 INFO:teuthology.kill:No teuthology processes running
2021-11-19T11:43:11.099 INFO:teuthology.kill:Nuking machines: ['smithi079', 'smithi194']
2021-11-19T11:43:12.074 INFO:teuthology.kill:2021-11-19 11:43:12,074.074 INFO:teuthology.nuke:targets:
2021-11-19T11:43:12.075 INFO:teuthology.kill:  smithi079.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpc6dt3fF4ARD5bfjiRodG2LJKBb4WKjsx5vBz4uTdtqjdt/oiMAFq41PSOFp5UUTbndSNj8WBNEb8SKYkOa6zpI3USj1swIjze43u/dVjtwqxU2lA1801SRC3kWQqdU4jeyV5aYp0pWJGILLO+gbK+xKAQQzrFP14NQlVxE2I+1Qg/zGK0OuQXfotHmKn5gWcQ1R/c7/h11aT9n+tK79moQ0AbpizJirnsfVn6dcfB6UofaptY+46sZoKWrLAnAxGHdm2HN+ZP12vsBEt58lKXWs0U7T8WWnYRvIAwRSClMHbDxjcQUI5n3reXEEZtM/dzID4pSdIUu3ewHkbimDCr4qTq6hibb00Hn8/dvPo0pGr0goSIpPy4yut67k9nTM9GDNRk+hADGmNX2kVhcPzTM7s58wnKdmLA0LLFLE9exOQfSKrLzRZJIeUy0UnZdTJlEAILIhJpbdxEvb4haXiDHbbzNCEHvi7Olf+BDcptLK9yE1UgMy7eWOqftz5G70=
2021-11-19T11:43:12.076 INFO:teuthology.kill:  smithi194.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLVmvejYlk0vCo0tyD1WK0dKVzWra8Qe+pxWXAULuC6QrdY7ftBt32+xeJmWEMT8mX8/e6jo47r3MDlUXYwF1+96VkIHerMjTk5CkVlmTEeVk7JYf4bPhvvG7MLr6hV7VlEV9CBWf4ChSg6M4TIOArEhsmCoVs22WaFSmIUZ1npvRtJ384ldYIb25yWgej04SA6NaXYq0DzfOjvw5fYAZImiZzEBwd0fCzCFe4a+mqp4CwJLBemjahkXpEQ5oteJoGuT5nny+tPx81kAytStqlkAhGtpyl+kY0edZMYCS0Muq4Xh8qkuH6YqXLhLEeW8AQrHV/gadTtVX03/NsHnNZMZqofbSALFNNrZoM8ZGvMv39EpLNiIlDJDBTz0lNyrmpFPj5CzxmnKpm8ezGOE8EXnxlA2ornd5+fJMXVB9GaaSb0zACALnvukC0L9aMI8zv4R5VZKnKJZMEmZvomcbI1hhIGMbLZXxLNwFuxMe5Xbc+3Q14nWiaf5ZYkhYl91M=
2021-11-19T11:43:12.100 INFO:teuthology.kill:2021-11-19 11:43:12,100.100 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-19T11:43:12.103 INFO:teuthology.kill:2021-11-19 11:43:12,103.103 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-19T11:43:12.180 INFO:teuthology.kill:2021-11-19 11:43:12,179.179 INFO:teuthology.orchestra.console:Power off smithi194
2021-11-19T11:43:12.195 INFO:teuthology.kill:2021-11-19 11:43:12,195.195 INFO:teuthology.orchestra.console:Power off smithi079
2021-11-19T11:43:20.553 INFO:teuthology.kill:2021-11-19 11:43:20,552.552 INFO:teuthology.orchestra.console:Power off for smithi194 completed
2021-11-19T11:43:20.702 INFO:teuthology.kill:2021-11-19 11:43:20,701.701 INFO:teuthology.lock.ops:unlocked: smithi194.front.sepia.ceph.com
2021-11-19T11:43:24.673 INFO:teuthology.kill:2021-11-19 11:43:24,672.672 INFO:teuthology.orchestra.console:Power off for smithi079 completed
2021-11-19T11:43:24.822 INFO:teuthology.kill:2021-11-19 11:43:24,822.822 INFO:teuthology.lock.ops:unlocked: smithi079.front.sepia.ceph.com
2021-11-19T11:45:25.133 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-19T11:45:25.253 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi079 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-11-19_05:17:03-krbd-pacific-testing-default-smithi/6513164
2021-11-19T11:45:25.253 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi194 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-11-19_05:17:03-krbd-pacific-testing-default-smithi/6513164