2021-11-17T15:44:38.008 INFO:root:teuthology version: 1.1.0-6cfddd5e
2021-11-17T15:44:38.127 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi013.front.sepia.ceph.com' reimaging
2021-11-17T15:44:38.128 INFO:teuthology.lock.ops:Updating [ubuntu@smithi013.front.sepia.ceph.com]: reset os type and version on server
2021-11-17T15:44:38.129 INFO:teuthology.lock.ops:Updating smithi013.front.sepia.ceph.com on lock server
2021-11-17T15:44:38.148 INFO:teuthology.lock.ops:Node 'ubuntu@smithi013.front.sepia.ceph.com' reimaging is complete
2021-11-17T15:44:38.149 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging
2021-11-17T15:44:38.149 INFO:teuthology.lock.ops:Updating [ubuntu@smithi072.front.sepia.ceph.com]: reset os type and version on server
2021-11-17T15:44:38.149 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server
2021-11-17T15:44:38.170 INFO:teuthology.lock.ops:Node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging is complete
2021-11-17T15:44:38.361 INFO:teuthology.provision.fog.smithi013:Scheduling deploy of centos 8.3
2021-11-17T15:44:38.381 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of centos 8.3
2021-11-17T15:44:38.763 INFO:teuthology.orchestra.console:Power off smithi013
2021-11-17T15:44:38.804 INFO:teuthology.orchestra.console:Power off smithi072
2021-11-17T15:44:47.143 INFO:teuthology.orchestra.console:Power off for smithi013 completed
2021-11-17T15:44:47.244 INFO:teuthology.orchestra.console:Power on smithi013
2021-11-17T15:44:51.281 INFO:teuthology.orchestra.console:Power off for smithi072 completed
2021-11-17T15:44:51.383 INFO:teuthology.orchestra.console:Power on smithi072
2021-11-17T15:44:55.609 INFO:teuthology.orchestra.console:Power on for smithi013 completed
2021-11-17T15:44:55.637 INFO:teuthology.orchestra.console:Power on for smithi072 completed
2021-11-17T15:44:55.711 INFO:teuthology.provision.fog.smithi013:Waiting for deploy to finish
2021-11-17T15:44:55.738 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish
2021-11-17T15:49:30.394 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-17T15:49:30.844 INFO:teuthology.provision.fog.smithi013:Node is ready
2021-11-17T15:49:30.861 INFO:teuthology.orchestra.run.smithi013.stdout:smithi013.front.sepia.ceph.com
2021-11-17T15:49:30.917 INFO:teuthology.orchestra.run.smithi013.stdout:172.21.15.13 smithi013.front.sepia.ceph.com smithi013
2021-11-17T15:49:31.251 INFO:teuthology.provision.fog.smithi013:Deploy complete!
2021-11-17T15:50:15.927 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-17T15:50:16.302 INFO:teuthology.provision.fog.smithi072:Node is ready
2021-11-17T15:50:16.318 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072.front.sepia.ceph.com
2021-11-17T15:50:16.374 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072
2021-11-17T15:50:16.706 INFO:teuthology.provision.fog.smithi072:Deploy complete!
2021-11-17T15:50:16.872 INFO:teuthology.lock.ops:Checking smithi013.front.sepia.ceph.com
2021-11-17T15:50:16.873 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-17T15:50:16.894 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com
2021-11-17T15:50:16.894 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-17T15:50:16.914 INFO:teuthology.lock.ops:Updating [smithi013.front.sepia.ceph.com]: set os type and version on server
2021-11-17T15:50:16.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi013.front.sepia.ceph.com'
2021-11-17T15:50:17.194 INFO:teuthology.orchestra.run.smithi013.stdout:x86_64
2021-11-17T15:50:17.251 INFO:teuthology.orchestra.run.smithi013.stdout:NAME="CentOS Linux"
2021-11-17T15:50:17.252 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION="8"
2021-11-17T15:50:17.252 INFO:teuthology.orchestra.run.smithi013.stdout:ID="centos"
2021-11-17T15:50:17.252 INFO:teuthology.orchestra.run.smithi013.stdout:ID_LIKE="rhel fedora"
2021-11-17T15:50:17.253 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION_ID="8"
2021-11-17T15:50:17.253 INFO:teuthology.orchestra.run.smithi013.stdout:PLATFORM_ID="platform:el8"
2021-11-17T15:50:17.253 INFO:teuthology.orchestra.run.smithi013.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-17T15:50:17.253 INFO:teuthology.orchestra.run.smithi013.stdout:ANSI_COLOR="0;31"
2021-11-17T15:50:17.253 INFO:teuthology.orchestra.run.smithi013.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-17T15:50:17.254 INFO:teuthology.orchestra.run.smithi013.stdout:HOME_URL="https://centos.org/"
2021-11-17T15:50:17.254 INFO:teuthology.orchestra.run.smithi013.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-17T15:50:17.254 INFO:teuthology.orchestra.run.smithi013.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-17T15:50:17.255 INFO:teuthology.orchestra.run.smithi013.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-17T15:50:17.256 INFO:teuthology.lock.ops:Updating smithi013.front.sepia.ceph.com on lock server
2021-11-17T15:50:17.277 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server
2021-11-17T15:50:17.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com'
2021-11-17T15:50:17.479 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64
2021-11-17T15:50:17.534 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Linux"
2021-11-17T15:50:17.534 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8"
2021-11-17T15:50:17.535 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos"
2021-11-17T15:50:17.535 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora"
2021-11-17T15:50:17.535 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8"
2021-11-17T15:50:17.538 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8"
2021-11-17T15:50:17.538 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-17T15:50:17.538 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31"
2021-11-17T15:50:17.538 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-17T15:50:17.539 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/"
2021-11-17T15:50:17.539 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-17T15:50:17.539 INFO:teuthology.orchestra.run.smithi072.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-17T15:50:17.539 INFO:teuthology.orchestra.run.smithi072.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-17T15:50:17.540 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server
2021-11-17T15:50:17.651 INFO:teuthology.dispatcher.supervisor:Running job 6509818
2021-11-17T15:50:17.652 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-17_15:36:30-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6509818 --name yuriw-2021-11-17_15:36:30-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2021-11-17_15:36:30-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6509818/orig.config.yaml
2021-11-17T15:50:17.658 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-11-17_15:36:30-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6509818
2021-11-17T15:50:17.658 INFO:teuthology.dispatcher.supervisor:Job PID: 14429
2021-11-17T15:50:17.659 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-18T03:51:11.923 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-18T03:51:12.129 INFO:teuthology.kill:Killing Pids: {14429}
2021-11-18T03:51:13.468 INFO:teuthology.task.internal:roles: ubuntu@smithi013.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-18T03:51:13.469 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-18T03:51:13.469 INFO:teuthology.misc:Compressing logs...
2021-11-18T03:51:13.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi013.front.sepia.ceph.com'
2021-11-18T03:51:13.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com'
2021-11-18T03:51:14.041 INFO:teuthology.misc:Compressing logs...
2021-11-18T03:51:14.081 INFO:teuthology.orchestra.run.smithi013.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-18T03:51:14.104 INFO:teuthology.orchestra.run.smithi072.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-18T03:51:14.150 INFO:teuthology.orchestra.run.smithi013.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-18T03:51:14.150 INFO:teuthology.orchestra.run.smithi013.stderr:tar: Error is not recoverable: exiting now
2021-11-18T03:51:14.174 INFO:teuthology.orchestra.run.smithi072.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-18T03:51:14.175 INFO:teuthology.orchestra.run.smithi072.stderr:tar: Error is not recoverable: exiting now
2021-11-18T03:51:14.176 INFO:teuthology.misc:Compressing logs...
2021-11-18T03:51:14.219 INFO:teuthology.orchestra.run.smithi013.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-18T03:51:14.221 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi013 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-18T03:51:14.292 INFO:teuthology.kill:No teuthology processes running
2021-11-18T03:51:15.444 INFO:teuthology.kill:Nuking machines: ['smithi013', 'smithi072']
2021-11-18T03:51:16.406 INFO:teuthology.kill:2021-11-18 03:51:16,406.406 INFO:teuthology.nuke:targets:
2021-11-18T03:51:16.407 INFO:teuthology.kill:  smithi013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN97PTTfCRAQ124N1DFX7B0wZEaDCgBL2Fi0IN4hQOQnyapSOCmcsynBMiCPqUfU0cYQj42kYeeovRlYHw0644x0gXO7gDnr8HcihR0UiimuTfcK2WCJWMZUMk+9RQ31C8yNWfh5ivg03H5D6XwZEZlCkDVSSiVwCtGdrTz9JbjtiS9FBzn9QQvxK9vtBSvUhdimUJZ8Vwic7ZXdNigVyeFXPcRomH07yRrA7PT87aR+Rs5xSYJ+XO/+r8KNggztSS9xxGtPNa9VRYctCJ1mgQxKYMBvu8fvffKlJN+Topwu64Pcjzbu0EbBgOvF4IEFifgKA7UU8thFqRs/XyBwIBnMdy3BXXw1Q8hmK2Gnny5ykTvsmzG72m3rq4QD5zNOqPCKZ4fV4I2U7JDP//aDu4WYL5ChH8tcX0I4x/yd3MrcYnRus1YpDXP3x7mC35t3+DlTb/kg6Uovw9Sy04617vi9/BDHYkvJkqU4xoynuzE4Dll8pNTssdOlGCk9vTOYc=
2021-11-18T03:51:16.408 INFO:teuthology.kill:  smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhqlNpFHi8mMmuXt6UYVhBeZaYffXHp97MVFrnqzQwXC+Bie1TtJuIRsxuu9pnLg4l6mouxSUYM8jIpwwK+Sc2oFJ18ATqWYaunYBrLRIHq+GOYF+CmxKsteeXZfAVNvm35aVAe21fiwdPRf2A3IiH9iUpxrbT+nVb/MfaJuIR96S3dL7tgx1MWkImpIvO/ka3FDoFv6lbzIoxrBQWWnKSSrYJuJ1yIGAYwr8uIgGZ5H/u8f/BGwVe/uOEP2ZPLtt1f4aEJPB+I+vEJIi5MVXUsZKL4v0zem9+VFS+bkHrUuRNuwwoVjd03U+0nOxXsHe0+oHIU5qmTzKVP9+SjquC/ZyPbWDvIlGj1v7MU5HcFo+sRiFrCvGspHtX533yym1tx3jSREfh2pIZIK+Tv28SGgFCpil6s8s9JMKxFTuC/tRBwgCQfhE/mYgz2ZrEqSlQcFNtBdtW7Zy62pMzOxIy0h3EBEkl+gMPuWuBDf7VklWYgX3FpQITeurheWd6iUU=
2021-11-18T03:51:16.432 INFO:teuthology.kill:2021-11-18 03:51:16,432.432 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-18T03:51:16.435 INFO:teuthology.kill:2021-11-18 03:51:16,435.435 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-18T03:51:16.510 INFO:teuthology.kill:2021-11-18 03:51:16,510.510 INFO:teuthology.orchestra.console:Power off smithi072
2021-11-18T03:51:16.525 INFO:teuthology.kill:2021-11-18 03:51:16,524.524 INFO:teuthology.orchestra.console:Power off smithi013
2021-11-18T03:51:24.877 INFO:teuthology.kill:2021-11-18 03:51:24,877.877 INFO:teuthology.orchestra.console:Power off for smithi072 completed
2021-11-18T03:51:24.886 INFO:teuthology.kill:2021-11-18 03:51:24,886.886 INFO:teuthology.orchestra.console:Power off for smithi013 completed
2021-11-18T03:51:25.025 INFO:teuthology.kill:2021-11-18 03:51:25,025.025 INFO:teuthology.lock.ops:unlocked: smithi072.front.sepia.ceph.com
2021-11-18T03:51:25.029 INFO:teuthology.kill:2021-11-18 03:51:25,029.029 INFO:teuthology.lock.ops:unlocked: smithi013.front.sepia.ceph.com
2021-11-18T03:53:25.329 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-18T03:53:25.448 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi013 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-18_00:45:42-rados-wip-yuri11-testing-2021-11-17-1514-pacific-distro-basic-smithi/6510981
2021-11-18T03:53:25.449 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi072 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-18_00:45:42-rados-wip-yuri11-testing-2021-11-17-1514-pacific-distro-basic-smithi/6510981