2021-11-16T22:06:26.730 INFO:root:teuthology version: 1.1.0-6cfddd5e
2021-11-16T22:06:26.860 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi083.front.sepia.ceph.com' reimaging
2021-11-16T22:06:26.862 INFO:teuthology.lock.ops:Updating [ubuntu@smithi083.front.sepia.ceph.com]: reset os type and version on server
2021-11-16T22:06:26.862 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server
2021-11-16T22:06:26.883 INFO:teuthology.lock.ops:Node 'ubuntu@smithi083.front.sepia.ceph.com' reimaging is complete
2021-11-16T22:06:26.884 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi134.front.sepia.ceph.com' reimaging
2021-11-16T22:06:26.884 INFO:teuthology.lock.ops:Updating [ubuntu@smithi134.front.sepia.ceph.com]: reset os type and version on server
2021-11-16T22:06:26.884 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server
2021-11-16T22:06:26.904 INFO:teuthology.lock.ops:Node 'ubuntu@smithi134.front.sepia.ceph.com' reimaging is complete
2021-11-16T22:06:27.088 INFO:teuthology.provision.fog.smithi083:Scheduling deploy of centos 8.3
2021-11-16T22:06:27.110 INFO:teuthology.provision.fog.smithi134:Scheduling deploy of centos 8.3
2021-11-16T22:06:27.439 INFO:teuthology.orchestra.console:Power off smithi083
2021-11-16T22:06:27.473 INFO:teuthology.orchestra.console:Power off smithi134
2021-11-16T22:06:35.835 INFO:teuthology.orchestra.console:Power off for smithi134 completed
2021-11-16T22:06:35.936 INFO:teuthology.orchestra.console:Power on smithi134
2021-11-16T22:06:39.923 INFO:teuthology.orchestra.console:Power off for smithi083 completed
2021-11-16T22:06:40.025 INFO:teuthology.orchestra.console:Power on smithi083
2021-11-16T22:06:44.295 INFO:teuthology.orchestra.console:Power on for smithi134 completed
2021-11-16T22:06:44.396 INFO:teuthology.provision.fog.smithi134:Waiting for deploy to finish
2021-11-16T22:06:48.394 INFO:teuthology.orchestra.console:Power on for smithi083 completed
2021-11-16T22:06:48.495 INFO:teuthology.provision.fog.smithi083:Waiting for deploy to finish
2021-11-16T22:11:34.213 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-16T22:11:34.503 INFO:teuthology.provision.fog.smithi134:Node is ready
2021-11-16T22:11:34.519 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134.front.sepia.ceph.com
2021-11-16T22:11:34.574 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134 smithi134.front.sepia.ceph.com smithi134
2021-11-16T22:11:34.923 INFO:teuthology.provision.fog.smithi134:Deploy complete!
2021-11-16T22:11:38.439 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-16T22:11:38.722 INFO:teuthology.provision.fog.smithi083:Node is ready
2021-11-16T22:11:38.737 INFO:teuthology.orchestra.run.smithi083.stdout:smithi083.front.sepia.ceph.com
2021-11-16T22:11:38.793 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83 smithi083.front.sepia.ceph.com smithi083
2021-11-16T22:11:39.117 INFO:teuthology.provision.fog.smithi083:Deploy complete!
2021-11-16T22:11:39.271 INFO:teuthology.lock.ops:Checking smithi083.front.sepia.ceph.com
2021-11-16T22:11:39.272 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-16T22:11:39.291 INFO:teuthology.lock.ops:Checking smithi134.front.sepia.ceph.com
2021-11-16T22:11:39.291 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-16T22:11:39.311 INFO:teuthology.lock.ops:Updating [smithi083.front.sepia.ceph.com]: set os type and version on server
2021-11-16T22:11:39.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com'
2021-11-16T22:11:39.564 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64
2021-11-16T22:11:39.618 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Linux"
2021-11-16T22:11:39.618 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="8"
2021-11-16T22:11:39.618 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos"
2021-11-16T22:11:39.619 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora"
2021-11-16T22:11:39.619 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="8"
2021-11-16T22:11:39.619 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el8"
2021-11-16T22:11:39.620 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-16T22:11:39.620 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31"
2021-11-16T22:11:39.620 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-16T22:11:39.620 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/"
2021-11-16T22:11:39.621 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-16T22:11:39.621 INFO:teuthology.orchestra.run.smithi083.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-16T22:11:39.621 INFO:teuthology.orchestra.run.smithi083.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-16T22:11:39.622 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server
2021-11-16T22:11:39.640 INFO:teuthology.lock.ops:Updating [smithi134.front.sepia.ceph.com]: set os type and version on server
2021-11-16T22:11:39.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com'
2021-11-16T22:11:39.911 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64
2021-11-16T22:11:39.965 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Linux"
2021-11-16T22:11:39.966 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="8"
2021-11-16T22:11:39.966 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos"
2021-11-16T22:11:39.966 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora"
2021-11-16T22:11:39.967 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="8"
2021-11-16T22:11:39.967 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el8"
2021-11-16T22:11:39.967 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-16T22:11:39.968 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31"
2021-11-16T22:11:39.968 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-16T22:11:39.969 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/"
2021-11-16T22:11:39.969 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-16T22:11:39.969 INFO:teuthology.orchestra.run.smithi134.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-16T22:11:39.970 INFO:teuthology.orchestra.run.smithi134.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-16T22:11:39.971 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server
2021-11-16T22:11:40.077 INFO:teuthology.dispatcher.supervisor:Running job 6507510
2021-11-16T22:11:40.077 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-16_20:03:49-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6507510 --name yuriw-2021-11-16_20:03:49-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2021-11-16_20:03:49-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6507510/orig.config.yaml
2021-11-16T22:11:40.081 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-11-16_20:03:49-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6507510
2021-11-16T22:11:40.082 INFO:teuthology.dispatcher.supervisor:Job PID: 17780
2021-11-16T22:11:40.082 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-17T10:12:35.187 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-17T10:12:35.405 INFO:teuthology.kill:Killing Pids: {17780}
2021-11-17T10:12:36.663 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-17T10:12:36.664 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-17T10:12:36.665 INFO:teuthology.misc:Compressing logs...
2021-11-17T10:12:36.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com'
2021-11-17T10:12:36.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com'
2021-11-17T10:12:37.420 INFO:teuthology.misc:Compressing logs...
2021-11-17T10:12:37.459 INFO:teuthology.orchestra.run.smithi083.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-17T10:12:37.483 INFO:teuthology.orchestra.run.smithi134.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-17T10:12:37.525 INFO:teuthology.orchestra.run.smithi083.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-17T10:12:37.525 INFO:teuthology.orchestra.run.smithi083.stderr:tar: Error is not recoverable: exiting now
2021-11-17T10:12:37.549 INFO:teuthology.orchestra.run.smithi134.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-17T10:12:37.549 INFO:teuthology.orchestra.run.smithi134.stderr:tar: Error is not recoverable: exiting now
2021-11-17T10:12:37.550 INFO:teuthology.misc:Compressing logs...
2021-11-17T10:12:37.594 INFO:teuthology.orchestra.run.smithi083.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-17T10:12:37.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-17T10:12:37.670 INFO:teuthology.kill:No teuthology processes running
2021-11-17T10:12:38.848 INFO:teuthology.kill:Nuking machines: ['smithi083', 'smithi134']
2021-11-17T10:12:39.905 INFO:teuthology.kill:2021-11-17 10:12:39,904.904 INFO:teuthology.nuke:targets:
2021-11-17T10:12:39.906 INFO:teuthology.kill:  smithi083.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDErqv+NPA+Qxvy2zdKZVlHbXU5B+R2aMoDFHshu10AtJ2zAJNbiijuK2D/aOG0vO5jDNgJlbAjWTLvWMC9IyxqrhxKJorGrjydGdhiKPwUClsfRu+XTRhIGF0/cb/uAaQ9Kx1WHtu14rPsyK9eELgo1Z6mWUAWZqzTIIiduNanoY34h8N4SBlRjo+nac/dNqoA8G4qvEuqE4S5hDaCO4jPOtfJ2hHUv5cD9jAZwezuheayBbjj1kxv51WOlbNY8GoTBBOFrflMX1inWHFJ0IStVBT38/qrli+Rpcg5b5V//QLLn5Jh9kIh1RRWmY6na97Kx1/LyuND4MAuf+OYeB6qf15xXOkUO0RPXzs1LSGNEHsJGtKj6GfYGAbmretKJsX1x23PR75Yv08xvEy6gLn8sfwNo9MEzf0hwDqmsksMVG1fPMRUpAkizeIWR2TCDRfBuIoXjJOtjl9c+ohSrr+GR0QQdD4lk4jdfwIbhxXU/tCWJ4aYxCqAAo3/rJwXv3U=
2021-11-17T10:12:39.907 INFO:teuthology.kill:  smithi134.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHYAILCD1PzVWgdpvaFmGt+KRbBtHOp4s5Nf5JlhpnBbyGg26uw/Kqmje5yD1SG5KW82mVBZOZd2PleHnu1vlqPNMd9VQ4XRPhR/d6qn37elyVMVgeESYu4WZg8sC9hp3NhYvReGBp1viMHVGU60zN/+z7Vd/DNVdGre7FgRQiGKhHd3RneyTTAxYSrgMak7pGapqZ+fPlQeMh/AnRK5f3vmTMv730Ftu9FI/XslNVdkuxfpE3qAtqCng1GfTuB9HzJd6Mpn7v/ezc0COqphUBnmCdGyJASXfPXaZcNjElU+2VcbJxIFXt7+w/NrASAhriccpFc2rN9vZDMZ5+mSWL47mYPNaKh2puPgM7eLIj45cX928HMeyNh7D6V8QJgwlEDKkc0CYQEmfIzvMjTleTjnm3PM0J7/B0GPgP8aORwUpa8ga/14iyCVnHeoMuVjPjcPUI7j2slAmLwGyCgp+EH/YaS3XmfYRlzsJn3wtdQhkUV+bPFWfsBCHT/J/1P2U=
2021-11-17T10:12:39.930 INFO:teuthology.kill:2021-11-17 10:12:39,930.930 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-17T10:12:39.936 INFO:teuthology.kill:2021-11-17 10:12:39,935.935 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-17T10:12:40.016 INFO:teuthology.kill:2021-11-17 10:12:40,016.016 INFO:teuthology.orchestra.console:Power off smithi134
2021-11-17T10:12:40.035 INFO:teuthology.kill:2021-11-17 10:12:40,035.035 INFO:teuthology.orchestra.console:Power off smithi083
2021-11-17T10:12:48.393 INFO:teuthology.kill:2021-11-17 10:12:48,393.393 INFO:teuthology.orchestra.console:Power off for smithi134 completed
2021-11-17T10:12:48.407 INFO:teuthology.kill:2021-11-17 10:12:48,407.407 INFO:teuthology.orchestra.console:Power off for smithi083 completed
2021-11-17T10:12:48.544 INFO:teuthology.kill:2021-11-17 10:12:48,543.543 INFO:teuthology.lock.ops:unlocked: smithi134.front.sepia.ceph.com
2021-11-17T10:12:48.555 INFO:teuthology.kill:2021-11-17 10:12:48,555.555 INFO:teuthology.lock.ops:unlocked: smithi083.front.sepia.ceph.com
2021-11-17T10:14:48.893 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-17T10:14:49.003 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi083 but it was locked by another job: /home/teuthworker/archive/pdonnell-2021-11-16_22:20:34-fs-wip-pdonnell-testing-20211116.201517-distro-basic-smithi/6508106
2021-11-17T10:14:49.003 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi134 but it was locked by another job: /home/teuthworker/archive/pdonnell-2021-11-16_22:20:34-fs-wip-pdonnell-testing-20211116.201517-distro-basic-smithi/6508106