2021-11-25T10:50:56.737 INFO:root:teuthology version: 1.1.0-a0bc3188
2021-11-25T10:50:56.861 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi027.front.sepia.ceph.com' reimaging
2021-11-25T10:50:56.862 INFO:teuthology.lock.ops:Updating [ubuntu@smithi027.front.sepia.ceph.com]: reset os type and version on server
2021-11-25T10:50:56.863 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2021-11-25T10:50:56.883 INFO:teuthology.lock.ops:Node 'ubuntu@smithi027.front.sepia.ceph.com' reimaging is complete
2021-11-25T10:50:56.883 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging
2021-11-25T10:50:56.883 INFO:teuthology.lock.ops:Updating [ubuntu@smithi029.front.sepia.ceph.com]: reset os type and version on server
2021-11-25T10:50:56.884 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2021-11-25T10:50:56.904 INFO:teuthology.lock.ops:Node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging is complete
2021-11-25T10:50:57.152 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of centos 8.3
2021-11-25T10:50:57.158 INFO:teuthology.provision.fog.smithi027:Scheduling deploy of centos 8.3
2021-11-25T10:50:57.682 INFO:teuthology.orchestra.console:Power off smithi027
2021-11-25T10:50:57.719 INFO:teuthology.orchestra.console:Power off smithi029
2021-11-25T10:50:57.829 INFO:teuthology.orchestra.console:Power off for smithi027 completed
2021-11-25T10:50:57.864 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2021-11-25T10:50:57.931 INFO:teuthology.orchestra.console:Power on smithi027
2021-11-25T10:50:57.965 INFO:teuthology.orchestra.console:Power on smithi029
2021-11-25T10:51:02.193 INFO:teuthology.orchestra.console:Power on for smithi027 completed
2021-11-25T10:51:02.219 INFO:teuthology.orchestra.console:Power on for smithi029 completed
2021-11-25T10:51:02.295 INFO:teuthology.provision.fog.smithi027:Waiting for deploy to finish
2021-11-25T10:51:02.322 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish
2021-11-25T10:55:37.700 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-25T10:55:37.994 INFO:teuthology.provision.fog.smithi027:Node is ready
2021-11-25T10:55:38.010 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027.front.sepia.ceph.com
2021-11-25T10:55:38.067 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27 smithi027.front.sepia.ceph.com smithi027
2021-11-25T10:55:38.432 INFO:teuthology.provision.fog.smithi027:Deploy complete!
2021-11-25T10:56:23.533 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-25T10:56:23.909 INFO:teuthology.provision.fog.smithi029:Node is ready
2021-11-25T10:56:23.928 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029.front.sepia.ceph.com
2021-11-25T10:56:23.989 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29 smithi029.front.sepia.ceph.com smithi029
2021-11-25T10:56:24.348 INFO:teuthology.provision.fog.smithi029:Deploy complete!
2021-11-25T10:56:24.576 INFO:teuthology.lock.ops:Checking smithi027.front.sepia.ceph.com
2021-11-25T10:56:24.577 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-25T10:56:24.602 INFO:teuthology.lock.ops:Checking smithi029.front.sepia.ceph.com
2021-11-25T10:56:24.602 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-25T10:56:24.629 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: set os type and version on server
2021-11-25T10:56:24.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2021-11-25T10:56:24.970 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64
2021-11-25T10:56:25.029 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Linux"
2021-11-25T10:56:25.030 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8"
2021-11-25T10:56:25.030 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos"
2021-11-25T10:56:25.030 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora"
2021-11-25T10:56:25.031 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8"
2021-11-25T10:56:25.031 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8"
2021-11-25T10:56:25.032 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-25T10:56:25.032 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31"
2021-11-25T10:56:25.033 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-25T10:56:25.034 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/"
2021-11-25T10:56:25.034 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-25T10:56:25.034 INFO:teuthology.orchestra.run.smithi027.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-25T10:56:25.035 INFO:teuthology.orchestra.run.smithi027.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-25T10:56:25.036 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2021-11-25T10:56:25.057 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: set os type and version on server
2021-11-25T10:56:25.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2021-11-25T10:56:25.269 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64
2021-11-25T10:56:25.326 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Linux"
2021-11-25T10:56:25.327 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8"
2021-11-25T10:56:25.327 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos"
2021-11-25T10:56:25.327 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora"
2021-11-25T10:56:25.327 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8"
2021-11-25T10:56:25.328 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8"
2021-11-25T10:56:25.328 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-25T10:56:25.328 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31"
2021-11-25T10:56:25.329 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-25T10:56:25.329 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/"
2021-11-25T10:56:25.329 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-25T10:56:25.329 INFO:teuthology.orchestra.run.smithi029.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-25T10:56:25.330 INFO:teuthology.orchestra.run.smithi029.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-25T10:56:25.331 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2021-11-25T10:56:25.430 INFO:teuthology.dispatcher.supervisor:Running job 6525076
2021-11-25T10:56:25.430 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_a0bc318848afe71da1fbf34e1e1b44738d8e3d6d/virtualenv/bin/teuthology -v --owner scheduled_ifed01@teuthology --archive /home/teuthworker/archive/ifed01-2021-11-25_10:30:40-rados-wip-ifed-better-daemonperf-distro-default-smithi/6525076 --name ifed01-2021-11-25_10:30:40-rados-wip-ifed-better-daemonperf-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/ifed01-2021-11-25_10:30:40-rados-wip-ifed-better-daemonperf-distro-default-smithi/6525076/orig.config.yaml
2021-11-25T10:56:25.434 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ifed01-2021-11-25_10:30:40-rados-wip-ifed-better-daemonperf-distro-default-smithi/6525076
2021-11-25T10:56:25.435 INFO:teuthology.dispatcher.supervisor:Job PID: 8154
2021-11-25T10:56:25.435 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-25T22:57:18.887 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-25T22:57:19.168 INFO:teuthology.kill:Killing Pids: {8154}
2021-11-25T22:57:20.401 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-25T22:57:20.402 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-25T22:57:20.403 INFO:teuthology.misc:Compressing logs...
2021-11-25T22:57:20.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2021-11-25T22:57:20.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2021-11-25T22:57:20.994 INFO:teuthology.misc:Compressing logs...
2021-11-25T22:57:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-25T22:57:21.059 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-25T22:57:21.100 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-25T22:57:21.100 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now
2021-11-25T22:57:21.128 INFO:teuthology.orchestra.run.smithi029.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-25T22:57:21.129 INFO:teuthology.orchestra.run.smithi029.stderr:tar: Error is not recoverable: exiting now
2021-11-25T22:57:21.131 INFO:teuthology.misc:Compressing logs...
2021-11-25T22:57:21.174 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-25T22:57:21.178 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a0bc318848afe71da1fbf34e1e1b44738d8e3d6d/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a0bc318848afe71da1fbf34e1e1b44738d8e3d6d/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a0bc318848afe71da1fbf34e1e1b44738d8e3d6d/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a0bc318848afe71da1fbf34e1e1b44738d8e3d6d/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a0bc318848afe71da1fbf34e1e1b44738d8e3d6d/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a0bc318848afe71da1fbf34e1e1b44738d8e3d6d/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-25T22:57:21.263 INFO:teuthology.kill:No teuthology processes running
2021-11-25T22:57:22.549 INFO:teuthology.kill:Nuking machines: ['smithi027', 'smithi029']
2021-11-25T22:57:23.542 INFO:teuthology.kill:2021-11-25 22:57:23,541.541 INFO:teuthology.nuke:targets:
2021-11-25T22:57:23.543 INFO:teuthology.kill:  smithi027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPzU0eESOxmXtfX9qigu/o4zDzT3OJR5xNgfpm/girr9BWOEC1fy/h4uSBQuy7lNdDPX4aS0636EZSgygZf4NdvSTecl3WBlNG6gJSPdTAbirKWfVckWTEQhlVW38ebOnvoo3KXLAURc/DuUDREiUZ5S/aKbLOqxP4D4eZJAwhdQhu76ZUScJHMaJ2CzEh8Fi6PgWSRKXGoTaxRyVpC5Mt5EjrS4gCtQ2481KqjkJRt8uXJzTqvmIWnkus8mHqt1uHjVbTknHSDa0O0qNu/1sG+mG8l9TrOFW/cgr61vKSIiXmQyyrhmVT8jGC2fJ4R3jz5CIV+0TjQyTEnNA4yTJnAZNmntxTGhUi1qw5zI+yegxGOKOKOnvcGxm3DryAqZoJVIR13CqaLk69k7H6P5A2Upc17p0z9u2bs0ktWUtNm+xtHPhbLcKd2gA3k+32q9P2tNZ5WAVpZPgPqIAEFjkJbaBz664zhFRHvF0yBXRhZRduBIuinWn/vMX1Y3J3nNc=
2021-11-25T22:57:23.543 INFO:teuthology.kill:  smithi029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnaGNVcaqM0UEuX4h2GLuabLg1O9Qorae/qGkb7FbTMPH4Dy+T94t6I1N0h9Fqrmked1bmx71yK61bCPisbY/2B92kNypswwEwiSQFR+adZSSUZbkOzM0r6aqYREbrd/piUSWMkJh/Xl/TGO5pDMVfgec4+ACJZvK8VhojY6RImMuq6GGhLgQmezgcqwzij4Kv5iuNg9ePF+0skJFjz5Q1BmCPB9J440Wl5CG7d01/RMVYJnXV3HIpGnCSpLUVysNuaH/DrII8Jl4MQqWwI/QDL8EUfF52hKctVL2YeoBNxU++NgI5sZvPtFezEMstJcjXo8bLl93BmOXP60835zXdMKkt/tOFWjVTCR76X0d3qUXImQ2Ch0aur0I1h0TApoXEVFjClluXlk+SSIUFcENlVtc7izz38L8Z9ILyTc+g9a+aZmpz55qFQVTSc9h7qVgdJUwDTuN8tfL9j196Z/08uH7NV3aceWu2xd0CSbqB/+dKFfAMrKKkRsZkrDFDPGM=
2021-11-25T22:57:23.570 INFO:teuthology.kill:2021-11-25 22:57:23,570.570 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-25T22:57:23.573 INFO:teuthology.kill:2021-11-25 22:57:23,573.573 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-25T22:57:23.657 INFO:teuthology.kill:2021-11-25 22:57:23,657.657 INFO:teuthology.orchestra.console:Power off smithi029
2021-11-25T22:57:23.674 INFO:teuthology.kill:2021-11-25 22:57:23,673.673 INFO:teuthology.orchestra.console:Power off smithi027
2021-11-25T22:57:32.039 INFO:teuthology.kill:2021-11-25 22:57:32,039.039 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2021-11-25T22:57:32.047 INFO:teuthology.kill:2021-11-25 22:57:32,046.046 INFO:teuthology.orchestra.console:Power off for smithi027 completed
2021-11-25T22:57:32.186 INFO:teuthology.kill:2021-11-25 22:57:32,185.185 INFO:teuthology.lock.ops:unlocked: smithi029.front.sepia.ceph.com
2021-11-25T22:57:32.193 INFO:teuthology.kill:2021-11-25 22:57:32,193.193 INFO:teuthology.lock.ops:unlocked: smithi027.front.sepia.ceph.com
2021-11-25T22:59:32.528 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-25T22:59:32.674 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi027 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-25_15:18:01-rados-pacific-16.2.7_RC1-distro-basic-smithi/6526015
2021-11-25T22:59:32.674 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi029 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-25_15:18:01-rados-pacific-16.2.7_RC1-distro-basic-smithi/6526015