Found 200 jobs Found 9 distinct failure reasons Crash: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi106 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1708eb058534abb565fa5e8d98e68c3390fa73ea TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' ceph version 16.0.0-3461-g1708eb0 (1708eb058534abb565fa5e8d98e68c3390fa73ea) pacific (dev) 1: (()+0x128a0) [0x7f72e30498a0] 2: (gsignal()+0xc7) [0x7f72e2141f47] 3: (abort()+0x141) [0x7f72e21438b1] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f72e372910a] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f72e3729294] 6: (CDir::mark_dirty(LogSegment*, unsigned long)+0x55) [0x55932b761715] 7: (CDentry::mark_dirty(unsigned long, LogSegment*)+0x6b) [0x55932b74e55b] 8: (CInode::mark_dirty(LogSegment*)+0x6f) [0x55932b79660f] 9: (CInode::pop_and_dirty_projected_inode(LogSegment*, boost::intrusive_ptr const&)+0x1aa) [0x55932b7b7cba] 10: (MDCache::rollback_uncommitted_fragments()+0x5fd) [0x55932b65e0dd] 11: (MDSRank::reopen_log()+0x14d) [0x55932b53536d] 12: (MDSRank::reconnect_start()+0x418) [0x55932b535b28] 13: (MDSRankDispatcher::handle_mds_map(boost::intrusive_ptr const&, MDSMap const&)+0x2003) [0x55932b546633] 14: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0xe61) [0x55932b51c0f1] 15: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0xb3) [0x55932b51d873] 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0xbd) [0x55932b51e52d] 17: (DispatchQueue::entry()+0x11ea) [0x7f72e391b79a] 18: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f72e39c7f8d] 19: (()+0x76db) [0x7f72e303e6db] 20: (clone()+0x3f) [0x7f72e2224a3f] 2 jobs: ['5230974', '5230827'] suites intersection: ['ceph-thrash/mds', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'fs/thrash/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_snaptests}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['ceph-thrash/mds', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'fs/thrash/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_snaptests}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_tell_session_drop (tasks.cephfs.test_sessionmap.TestSessionMap) 1 jobs: ['5230909'] suites: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/sessionmap/{sessionmap}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2020-07-16T15:26:44.407472+0000 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi068: (7459), after waiting 48.1803 seconds during MDS startup" in cluster log 2 jobs: ['5231008', '5230851'] suites intersection: ['conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['centos_latest', 'clusters/1-mds-4-client-coloc', 'clusters/fixed-2-ucephfs', 'conf/{client', 'fs/basic_functional/{begin', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'no_client_pidfile', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{rhel_8}', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/sessionmap/{sessionmap}}', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_dump_inode (tasks.cephfs.test_misc.TestMisc) 2 jobs: ['5230823', '5230924'] suites intersection: ['conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{frag_enable', 'tasks/cephfs_misc_tests}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'tasks/cephfs_misc_tests}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_basic (tasks.cephfs.test_dump_tree.TestDumpTree) 2 jobs: ['5230934', '5230836'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'tasks/asok_dump_tree}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{rhel_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/asok_dump_tree}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: {'Failure object was': {'smithi122.front.sepia.ceph.com': {'msg': 'Failed to download packages: Cannot download Packages/qemu-kvm-block-rbd-2.12.0-88.module_el8.1.0+297+df420408.3.x86_64.rpm: All mirrors were tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu-kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"} 1 jobs: ['5230933'] suites: ['clusters/fixed-2-ucephfs', 'conf/{client', 'fs/basic_workload/{begin', 'inline/yes', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/cfuse_workunit_norstats}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) 2 jobs: ['5230879', '5230978'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/fragment}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/fragment}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed on smithi008 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 2 jobs: ['5230949', '5230844'] suites intersection: ['conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'clusters/fixed-2-ucephfs', 'conf/{client', 'fs/basic_functional/{begin', 'fs/basic_workload/{begin', 'inline/no', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/cap-flush}', 'tasks/cfuse_workunit_suites_fsx}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_df_for_valid_file (tasks.cephfs.test_cephfs_shell.TestDF) 1 jobs: ['5230847'] suites: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cephfs-shell}', 'whitelist_health', 'whitelist_wrongly_marked_down}']