2021-11-10T16:51:51.371 INFO:root:teuthology version: 1.1.0-27954452
2021-11-10T16:51:51.492 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi042.front.sepia.ceph.com' reimaging
2021-11-10T16:51:51.494 INFO:teuthology.lock.ops:Updating [ubuntu@smithi042.front.sepia.ceph.com]: reset os type and version on server
2021-11-10T16:51:51.494 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server
2021-11-10T16:51:51.515 INFO:teuthology.lock.ops:Node 'ubuntu@smithi042.front.sepia.ceph.com' reimaging is complete
2021-11-10T16:51:51.516 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging
2021-11-10T16:51:51.516 INFO:teuthology.lock.ops:Updating [ubuntu@smithi114.front.sepia.ceph.com]: reset os type and version on server
2021-11-10T16:51:51.516 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2021-11-10T16:51:51.536 INFO:teuthology.lock.ops:Node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging is complete
2021-11-10T16:51:51.744 INFO:teuthology.provision.fog.smithi042:Scheduling deploy of centos 8.stream
2021-11-10T16:51:51.748 INFO:teuthology.provision.fog.smithi114:Scheduling deploy of centos 8.stream
2021-11-10T16:51:52.068 INFO:teuthology.orchestra.console:Power off smithi042
2021-11-10T16:51:52.084 INFO:teuthology.orchestra.console:Power off smithi114
2021-11-10T16:51:52.260 INFO:teuthology.orchestra.console:Power off for smithi042 completed
2021-11-10T16:51:52.341 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2021-11-10T16:51:52.362 INFO:teuthology.orchestra.console:Power on smithi042
2021-11-10T16:51:52.442 INFO:teuthology.orchestra.console:Power on smithi114
2021-11-10T16:51:56.697 INFO:teuthology.orchestra.console:Power on for smithi042 completed
2021-11-10T16:51:56.799 INFO:teuthology.provision.fog.smithi042:Waiting for deploy to finish
2021-11-10T16:51:56.918 INFO:teuthology.orchestra.console:Power on for smithi114 completed
2021-11-10T16:51:57.019 INFO:teuthology.provision.fog.smithi114:Waiting for deploy to finish
2021-11-10T16:54:12.906 ERROR:teuthology.orchestra.connection:Error authenticating with smithi042.front.sepia.ceph.com: Authentication failed.
2021-11-10T16:56:29.247 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-10T16:56:29.550 INFO:teuthology.provision.fog.smithi042:Node is ready
2021-11-10T16:56:29.568 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042.front.sepia.ceph.com
2021-11-10T16:56:29.627 INFO:teuthology.orchestra.run.smithi042.stdout:172.21.15.42 smithi042.front.sepia.ceph.com smithi042
2021-11-10T16:56:29.986 INFO:teuthology.provision.fog.smithi042:Deploy complete!
2021-11-10T16:57:01.418 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-10T16:57:01.705 INFO:teuthology.provision.fog.smithi114:Node is ready
2021-11-10T16:57:01.723 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114.front.sepia.ceph.com
2021-11-10T16:57:01.780 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114 smithi114.front.sepia.ceph.com smithi114
2021-11-10T16:57:02.132 INFO:teuthology.provision.fog.smithi114:Deploy complete!
2021-11-10T16:57:02.322 INFO:teuthology.lock.ops:Checking smithi042.front.sepia.ceph.com
2021-11-10T16:57:02.323 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-10T16:57:02.348 INFO:teuthology.lock.ops:Checking smithi114.front.sepia.ceph.com
2021-11-10T16:57:02.348 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-10T16:57:02.370 INFO:teuthology.lock.ops:Updating [smithi042.front.sepia.ceph.com]: set os type and version on server
2021-11-10T16:57:02.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com'
2021-11-10T16:57:02.616 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64
2021-11-10T16:57:02.672 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="CentOS Stream"
2021-11-10T16:57:02.673 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="8"
2021-11-10T16:57:02.673 INFO:teuthology.orchestra.run.smithi042.stdout:ID="centos"
2021-11-10T16:57:02.673 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE="rhel fedora"
2021-11-10T16:57:02.673 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="8"
2021-11-10T16:57:02.674 INFO:teuthology.orchestra.run.smithi042.stdout:PLATFORM_ID="platform:el8"
2021-11-10T16:57:02.674 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="CentOS Stream 8"
2021-11-10T16:57:02.674 INFO:teuthology.orchestra.run.smithi042.stdout:ANSI_COLOR="0;31"
2021-11-10T16:57:02.674 INFO:teuthology.orchestra.run.smithi042.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-10T16:57:02.675 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://centos.org/"
2021-11-10T16:57:02.675 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-11-10T16:57:02.675 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-11-10T16:57:02.675 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-11-10T16:57:02.677 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server
2021-11-10T16:57:02.696 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: set os type and version on server
2021-11-10T16:57:02.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2021-11-10T16:57:02.921 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64
2021-11-10T16:57:02.975 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream"
2021-11-10T16:57:02.976 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8"
2021-11-10T16:57:02.976 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos"
2021-11-10T16:57:02.976 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora"
2021-11-10T16:57:02.976 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8"
2021-11-10T16:57:02.977 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8"
2021-11-10T16:57:02.977 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 8"
2021-11-10T16:57:02.977 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31"
2021-11-10T16:57:02.977 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-10T16:57:02.978 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/"
2021-11-10T16:57:02.978 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-11-10T16:57:02.978 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-11-10T16:57:02.978 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-11-10T16:57:02.979 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2021-11-10T16:57:03.090 INFO:teuthology.dispatcher.supervisor:Running job 6496405
2021-11-10T16:57:03.090 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-11-10_16:44:53-orch:cephadm-master-distro-basic-smithi/6496405 --name sage-2021-11-10_16:44:53-orch:cephadm-master-distro-basic-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/sage-2021-11-10_16:44:53-orch:cephadm-master-distro-basic-smithi/6496405/orig.config.yaml
2021-11-10T16:57:03.094 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-11-10_16:44:53-orch:cephadm-master-distro-basic-smithi/6496405
2021-11-10T16:57:03.095 INFO:teuthology.dispatcher.supervisor:Job PID: 687
2021-11-10T16:57:03.095 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-11T04:57:23.208 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-11T04:57:23.447 INFO:teuthology.kill:Killing Pids: {687}
2021-11-11T04:57:24.663 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-11T04:57:24.664 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-11T04:57:24.665 INFO:teuthology.misc:Compressing logs...
2021-11-11T04:57:24.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com'
2021-11-11T04:57:24.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2021-11-11T04:57:25.197 INFO:teuthology.misc:Compressing logs...
2021-11-11T04:57:25.238 INFO:teuthology.orchestra.run.smithi042.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-11T04:57:25.262 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-11T04:57:25.309 INFO:teuthology.orchestra.run.smithi042.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-11T04:57:25.309 INFO:teuthology.orchestra.run.smithi042.stderr:tar: Error is not recoverable: exiting now
2021-11-11T04:57:25.334 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-11T04:57:25.335 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now
2021-11-11T04:57:25.336 INFO:teuthology.misc:Compressing logs...
2021-11-11T04:57:25.380 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-11T04:57:25.381 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi042 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-11T04:57:25.472 INFO:teuthology.kill:No teuthology processes running
2021-11-11T04:57:26.618 INFO:teuthology.kill:Nuking machines: ['smithi042', 'smithi114']
2021-11-11T04:57:27.569 INFO:teuthology.kill:2021-11-11 04:57:27,568.568 INFO:teuthology.nuke:targets:
2021-11-11T04:57:27.570 INFO:teuthology.kill:  smithi042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDU4oOo2/6873aKQ5BaZ05ZKWJTBEh1ia2fRFVUy2xqF3GKG8gwYwvLju4FjbZactT/O7UWck6PbavBWmYRKMUU7mxsTijfmg/G0cKTUyZHv8N/sk7rJBxBNgUlo2FsxFqiXh3a9ctzGEFTyui4XMhRhR4Z8l0cP2TSkpHj8PIgocc/db7FX71HlUwoG2FGR4y5eObOqg66BAHCdUpy/orrtOj0c+fBkdJo5Cu0ZlPUV9qcfUO+lCekfkAGbiXojxnPZaVLjEO44k4JmDyYGRlPmTZbgO4W/ih1oIkiis6YCdyleQnk2a6sOCZhbyz5eOg5rrdHT1LAa0exALTfen4rwOYOqs/OLefEwEdKzlR3qo2et3GVW4wt0Rvi2IHepymEew4U1EiH5d2V2IjyMpqQaBw/EvE+FFpRVo3Ua5GIf14YMVd/qxPrBszpzPXAAWDklxPTRDsEq/1c+RVgE45tTVTY7gcYT/+aTH1MNAodZZgs+g22qbBxzoObKQlMk/s=
2021-11-11T04:57:27.570 INFO:teuthology.kill:  smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLMCH3jFjjIAITTYKJxrLpMYVG5ZZSBrij3wd27MoNiqhPICmFvrF22Zeh70+HDc/CRkzFiEc4DIkf9LD3OMOpvjqrbETVW22h6KC9pWSvF+uEvxtb9+L6nZbaTAezxwgKO0Pe55faGw9zGAd96r3RF/x+TfHeZreswb8ZgKipc+yN+fj5Y5TyPb3c4M7t3B/DT1C587QCf/5kM3nvv6WoSJk/L0W7TKqos2lisEMOQ/RK/KzqhLz5QPl7zjJ+PMDSbKN+lsY6RVPBG4tSBXLToxu8XtuqC36Ld8XPpWlDp3IOci06oouAX9G8eiAJ3jvicS9NNhlFH5Piwwi6gJZNGoy19AcdcocFyi2v9eN1N6M3q2DQhThYfVQxzSc+wDGQ0Fd8NZmlcW9fIwDT60wyy/SPGoKY8zCNrctX6j4OdgWIvx/pQFjVuRiYQubALSSlMOidgsJVoLb0i3tQmv1vXhwDoPG8nwgMiZ+wg6GtvSc+QeD+oHfRRb+lKWa84vU=
2021-11-11T04:57:27.596 INFO:teuthology.kill:2021-11-11 04:57:27,596.596 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T04:57:27.599 INFO:teuthology.kill:2021-11-11 04:57:27,599.599 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T04:57:27.679 INFO:teuthology.kill:2021-11-11 04:57:27,679.679 INFO:teuthology.orchestra.console:Power off smithi114
2021-11-11T04:57:27.694 INFO:teuthology.kill:2021-11-11 04:57:27,694.694 INFO:teuthology.orchestra.console:Power off smithi042
2021-11-11T04:57:36.056 INFO:teuthology.kill:2021-11-11 04:57:36,055.055 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2021-11-11T04:57:36.210 INFO:teuthology.kill:2021-11-11 04:57:36,209.209 INFO:teuthology.lock.ops:unlocked: smithi114.front.sepia.ceph.com
2021-11-11T04:57:40.179 INFO:teuthology.kill:2021-11-11 04:57:40,178.178 INFO:teuthology.orchestra.console:Power off for smithi042 completed
2021-11-11T04:57:40.323 INFO:teuthology.kill:2021-11-11 04:57:40,323.323 INFO:teuthology.lock.ops:unlocked: smithi042.front.sepia.ceph.com
2021-11-11T04:59:40.557 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-11T04:59:40.659 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi042 but it was locked by another job: /home/teuthworker/archive/soumyakoduri-2021-11-11_04:48:08-rgw-cloudtiering-distro-basic-smithi/6496786
2021-11-11T04:59:40.659 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi114 but it was locked by another job: /home/teuthworker/archive/soumyakoduri-2021-11-11_04:48:08-rgw-cloudtiering-distro-basic-smithi/6496784