Found 93 jobs Missing teuthology log /home/teuthworker/archive/adking-2023-06-13_16:39:13-orch:cephadm-wip-adk-testing-2023-06-13-1036-distro-default-smithi/7302660/teuthology.log Found 5 distinct failure reasons Dead: b'2023-06-14T16:52:52.093 DEBUG:teuthology.exit:Finished running handlers' 14 jobs: ['7302650', '7302643', '7302652', '7302646', '7302639', '7302645', '7302653', '7302654', '7302651', '7302638', '7302647', '7302648', '7302644', '7302640'] suites intersection: [] suites union: ['0-nvme-loop', '1-start', '2-node-mgr', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-repo_digest/repo_digest', '2-services/basic', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '3-final}', '3-upgrade/simple', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v2only', 'openstack}', 'orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'orchestrator_cli}', 'start', 'start}', 'task/test_nfs}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'workloads/cephadm_iscsi}'] Failure: Command failed on smithi141 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6f56380bbd18e54ce5a7704e96f24c2ab58c9c0a pull' 1 jobs: ['7302665'] suites: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/connectivity', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'start}'] Failure: {'smithi152.front.sepia.ceph.com': {'_ansible_delegated_vars': {'ansible_connection': 'local', 'ansible_host': 'localhost', 'ansible_port': None, 'ansible_user': 'cm'}, '_ansible_no_log': False, 'attempts': 5, 'changed': False, 'invocation': {'module_args': {'accept_hostkey': False, 'archive': None, 'archive_prefix': None, 'bare': False, 'clone': True, 'depth': None, 'dest': '/home/teuthworker/.cache/src/keys', 'executable': None, 'force': True, 'gpg_whitelist': [], 'key_file': None, 'recursive': True, 'reference': None, 'refspec': None, 'remote': 'origin', 'repo': 'https://github.com/ceph/keys', 'separate_git_dir': None, 'ssh_opts': None, 'track_submodules': False, 'umask': None, 'update': True, 'verify_commit': False, 'version': 'main'}}, 'msg': 'Failed to checkout branch main', 'rc': 128, 'stderr': "fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n", 'stderr_lines': ["fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.", '', 'Another git process seems to be running in this repository, e.g.', "an editor opened by 'git commit'. Please make sure all processes", 'are terminated then try again. If it still fails, a git process', 'may have crashed in this repository earlier:', 'remove the file manually to continue.'], 'stdout': '', 'stdout_lines': []}} 1 jobs: ['7302641'] suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_monitoring_stack_basic}'] Failure: reached maximum tries (301) after waiting for 300 seconds 1 jobs: ['7302696'] suites: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun'] Dead: None 1 jobs: ['7302660']