Found 317 jobs Found 12 distinct failure reasons Failure: {'smithi114.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': 'failure writing module data to temporary file for transfer: [Errno 28] No space left on device'}} 1 jobs: ['6869535'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/radosbench}'] Failure: 'wait for operator' reached maximum tries (90) after waiting for 900 seconds 4 jobs: ['6869510', '6869353', '6869432', '6869590'] suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '3-final', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04'] suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}'] Failure: {'smithi155.front.sepia.ceph.com': {'_ansible_no_log': False, 'exception': 'Traceback (most recent call last):\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 124, in run\n item_results = self._run_loop(items)\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 357, in _run_loop\n res = self._execute(variables=task_vars)\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 663, in _execute\n result = self._handler.run(task_vars=variables)\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/plugins/action/normal.py", line 47, in run\n result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/plugins/action/__init__.py", line 861, in _execute_module\n (module_style, shebang, module_data, module_path) = self._configure_module(module_name=module_name, module_args=module_args, task_vars=task_vars)\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/plugins/action/__init__.py", line 236, in _configure_module\n **become_kwargs)\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/executor/module_common.py", line 1342, in modify_module\n environment=environment)\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/executor/module_common.py", line 1195, in _find_module_utils\n with open(cached_module_filename + \'-part\', \'wb\') as f:\nOSError: [Errno 28] No space left on device: \'/home/teuthworker/.ansible/tmp/ansible-local-29447bsmo9yho/ansiballz_cache/lvg-ZIP_DEFLATED-part\'\n', 'msg': 'Unexpected failure during module execution.', 'stdout': ''}} 10 jobs: ['6869539', '6869542', '6869521', '6869544', '6869546', '6869547', '6869543', '6869536', '6869541', '6869532'] suites intersection: ['rados'] suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'ceph', 'clusters/9-mons', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/default', 'd-balancer/on', 'd-thrash/none', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/monthrash/{ceph', 'rados/multimon/{clusters/6', 'rados/singleton-nomsgr/{all/balancer', 'rados/singleton-nomsgr/{all/ceph-post-file', 'rados/singleton/{all/erasure-code-nonregression', 'rados/singleton/{all/lost-unfound-delete', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'recovery-overrides/{default}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/mon_recovery', 'tasks/mon_recovery}', 'tasks/repair_test}', 'thrashers/default', 'thrashers/many', 'thrashosds-health', 'validater/lockdep}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/rados_mon_osdmap_prune}', 'workloads/redirect}'] Failure: Test failure: test_health_history (tasks.mgr.test_insights.TestInsights) 1 jobs: ['6869419'] suites: ['debug/mgr', 'mgr_ttl_cache/disable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-zlib}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/insights}'] Timeout 120 running ceph --cluster ceph osd dump --format=json 1 jobs: ['6869500'] suites: ['ceph', 'clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{default}', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] Dead: b'2022-06-09T08:07:57.693 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['6869392'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}'] Failure: {'smithi117.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': '/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000', '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'failed': True, 'invocation': {'content': 'CENSORED: content is a no_log parameter', 'dest': '/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt', 'module_args': {'content': 'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER', 'dest': '/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt'}}, 'item': '/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000', 'msg': "could not write content temp file: [Errno 28] No space left on device: '/home/teuthworker/.ansible/tmp/ansible-local-26963k5vebarb/tmpkzdx8n2l'"}, {'_ansible_item_label': '/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000', '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'checksum': 'bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79', 'dest': '/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt', 'diff': {'after': {'path': '/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt'}, 'before': {'path': '/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt'}}, 'failed': False, 'gid': 0, 'group': 'root', 'invocation': {'module_args': {'_diff_peek': None, '_original_basename': 'tmprakk4fve', 'access_time': None, 'access_time_format': '%Y%m%d%H%M.%S', 'attributes': None, 'dest': '/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt', 'follow': True, 'force': False, 'group': None, 'mode': None, 'modification_time': None, 'modification_time_format': '%Y%m%d%H%M.%S', 'owner': None, 'path': '/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt', 'recurse': False, 'selevel': None, 'serole': None, 'setype': None, 'seuser': None, 'src': None, 'state': 'file', 'unsafe_writes': False}}, 'item': '/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000', 'mode': '0644', 'owner': 'root', 'path': '/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt', 'secontext': 'system_u:object_r:etc_t:s0', 'size': 2334, 'state': 'file', 'uid': 0}]}} 1 jobs: ['6869537'] suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_8}}'] Failure: {'smithi111.front.sepia.ceph.com': {'_ansible_delegated_vars': {'ansible_connection': 'local', 'ansible_host': 'localhost', 'ansible_port': None, 'ansible_user': 'cm'}, '_ansible_no_log': False, 'attempts': 5, 'changed': False, 'invocation': {'module_args': {'accept_hostkey': False, 'archive': None, 'archive_prefix': None, 'bare': False, 'clone': True, 'depth': None, 'dest': '/home/teuthworker/.cache/src/keys', 'executable': None, 'force': True, 'gpg_whitelist': [], 'key_file': None, 'recursive': True, 'reference': None, 'refspec': None, 'remote': 'origin', 'repo': 'https://github.com/ceph/keys', 'separate_git_dir': None, 'ssh_opts': None, 'track_submodules': False, 'umask': None, 'update': True, 'verify_commit': False, 'version': 'main'}}, 'msg': 'Failed to checkout branch main', 'rc': 128, 'stderr': "fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n", 'stderr_lines': ["fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.", '', 'Another git process seems to be running in this repository, e.g.', "an editor opened by 'git commit'. Please make sure all processes", 'are terminated then try again. If it still fails, a git process', 'may have crashed in this repository earlier:', 'remove the file manually to continue.'], 'stdout': '', 'stdout_lines': []}, 'smithi038.front.sepia.ceph.com': {'_ansible_delegated_vars': {'ansible_connection': 'local', 'ansible_host': 'localhost', 'ansible_port': None, 'ansible_user': 'cm'}, '_ansible_no_log': False, 'attempts': 5, 'changed': False, 'invocation': {'module_args': {'accept_hostkey': False, 'archive': None, 'archive_prefix': None, 'bare': False, 'clone': True, 'depth': None, 'dest': '/home/teuthworker/.cache/src/keys', 'executable': None, 'force': True, 'gpg_whitelist': [], 'key_file': None, 'recursive': True, 'reference': None, 'refspec': None, 'remote': 'origin', 'repo': 'https://github.com/ceph/keys', 'separate_git_dir': None, 'ssh_opts': None, 'track_submodules': False, 'umask': None, 'update': True, 'verify_commit': False, 'version': 'main'}}, 'msg': 'Failed to checkout branch main', 'rc': 128, 'stderr': "fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n", 'stderr_lines': ["fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.", '', 'Another git process seems to be running in this repository, e.g.', "an editor opened by 'git commit'. Please make sure all processes", 'are terminated then try again. If it still fails, a git process', 'may have crashed in this repository earlier:', 'remove the file manually to continue.'], 'stdout': '', 'stdout_lines': []}} 1 jobs: ['6869420'] suites: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi090 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b591395bc273d0505c75a33641a3f15fb39abf84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2 jobs: ['6869588', '6869431'] suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}'] suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'random-objectstore$/{filestore-xfs}', 'tasks/e2e}'] Failure: {'smithi120.front.sepia.ceph.com': {'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}} 1 jobs: ['6869533'] suites: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/connectivity', 'rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0', 'start}'] Failure: {'smithi074.front.sepia.ceph.com': {'_ansible_no_log': False, 'exception': 'Traceback (most recent call last):\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 158, in run\n res = self._execute()\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 663, in _execute\n result = self._handler.run(task_vars=variables)\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_2290146eac7577b8500f128a53856c3ea4a00e3c/virtualenv/lib/python3.6/site-packages/ansible/plugins/action/template.py", line 157, in run\n local_tempdir = tempfile.mkdtemp(dir=C.DEFAULT_LOCAL_TMP)\n File "/usr/lib/python3.6/tempfile.py", line 509, in mkdtemp\n _os.mkdir(file, 0o700)\nOSError: [Errno 28] No space left on device: \'/home/teuthworker/.ansible/tmp/ansible-local-9487zujpu8al/tmpkof6fl_x\'\n', 'msg': 'Unexpected failure during module execution.', 'stdout': ''}, 'smithi100.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': 'failure writing module data to temporary file for transfer: [Errno 28] No space left on device'}} 1 jobs: ['6869528'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] Failure: Command failed (workunit test rados/test_python.sh) on smithi110 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b591395bc273d0505c75a33641a3f15fb39abf84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh' 1 jobs: ['6869375'] suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_python}']