Found 362 jobs Found 7 distinct failure reasons Failure: reached maximum tries (180) after waiting for 180 seconds 2 jobs: ['6015939', '6016074'] suites intersection: ['fixed-2', 'msgr/async-v1only'] suites union: ['1-start', '2-thrash', '3-tasks/snaps-few-objects', 'fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr/async-v1only', 'rados/cephadm/thrash/{0-distro/ubuntu_20.04_kubic_stable', 'rados/cephadm/with-work/{0-distro/ubuntu_20.04_kubic_stable', 'root}', 'start', 'tasks/rados_api_tests}'] Failure: Command failed on smithi132 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e010a7aeb144b1d2e29c547cd26d140834c7544e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d5c6c44-937b-11eb-8176-001a4aab830c -- ceph mon dump -f json' 2 jobs: ['6016220', '6016201'] suites intersection: [] suites union: ['1-start', '2-services/mirror', '3-final}', 'fixed-2', 'mon_election/classic', 'rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04_kubic_stable', 'rados/cephadm/smoke/{distro/ubuntu_20.04_kubic_stable', 'start}'] Failure: {'Failure object was': {'smithi013.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} 1 jobs: ['6015931'] suites: ['2-repo_digest/defaut', '3-start-upgrade', '4-wait', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] Dead: b'2021-04-02T17:11:13.309 DEBUG:teuthology.exit:Finished running handlers' ceph version 16.2.0-5-ge010a7ae (e010a7aeb144b1d2e29c547cd26d140834c7544e) pacific (stable) 1: /lib64/libpthread.so.0(+0x12dc0) [0x7f4523253dc0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55faf72cb509] 5: (KernelDevice::_aio_thread()+0x1254) [0x55faf7e09954] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55faf7e14c51] 7: /lib64/libpthread.so.0(+0x82de) [0x7f45232492de] 8: clone() 1 jobs: ['6016116'] suites: ['debug/mgr', 'mon_election/connectivity', 'objectstore/bluestore-hybrid', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{centos_8}', 'tasks/progress}'] Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 1 jobs: ['6016030'] suites: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'thrashers/many', 'workloads/rados_api_tests}'] Failure: SELinux denials found on ubuntu@smithi072.front.sepia.ceph.com: ['type=AVC msg=audit(1617344001.686:276): avc: denied { create } for pid=5888 comm="nm-dispatcher" name="1617344001.5888.core" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1617344001.686:275): avc: denied { add_name } for pid=5888 comm="nm-dispatcher" name="1617344001.5888.core" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1617344001.686:274): avc: denied { write } for pid=5888 comm="nm-dispatcher" name="coredump" dev="sda1" ino=394973 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1617344001.686:277): avc: denied { write } for pid=5888 comm="nm-dispatcher" path="/home/ubuntu/cephtest/archive/coredump/1617344001.5888.core" dev="sda1" ino=394296 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=1'] 1 jobs: ['6016216'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-snaps}'] Failure: Command failed (workunit test cephtool/test.sh) on smithi195 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e010a7aeb144b1d2e29c547cd26d140834c7544e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 1 jobs: ['6016156'] suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{rhel_8}}']