Found 230 jobs Not-understood crash indication Binary file /home/teuthworker/archive/pdonnell-2020-09-19_23:22:37-fs-wip-pdonnell-testing-20200919.211221-distro-basic-smithi/5448752/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/pdonnell-2020-09-19_23:22:37-fs-wip-pdonnell-testing-20200919.211221-distro-basic-smithi/5448752/teuthology.log matches Not-understood crash indication Found 8 distinct failure reasons Failure: Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2 jobs: ['5448918', '5448805'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/strays}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/strays}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi179 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fe900b354658252a2233b17cacef96f5e5d1fe30 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 5 jobs: ['5448706', '5448752', '5448844', '5448890', '5448798'] suites intersection: ['ceph-thrash/mds', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'fs/thrash/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'osd}', 'overrides/{frag_enable', 'tasks/cfuse_workunit_snaptests}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['ceph-thrash/mds', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'fs/thrash/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'supported-random-distros$/{rhel_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_snaptests}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) 2 jobs: ['5448758', '5448873'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'tasks/data-scan}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{rhel_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/data-scan}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_split_straydir (tasks.cephfs.test_fragment.TestFragmentation) 2 jobs: ['5448879', '5448764'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'tasks/fragment}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{rhel_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/fragment}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_standby_replay_singleton_fail_multimds (tasks.cephfs.test_failover.TestStandbyReplay) 1 jobs: ['5448749'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/failover}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Test failure: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) ceph version 16.0.0-5770-gfe900b35 (fe900b354658252a2233b17cacef96f5e5d1fe30) pacific (dev) 1: () 2: gsignal() 3: abort() 4: () 5: () 6: () 7: () 8: (ceph::buffer::v15_2_0::ptr::iterator_impl::operator+=(unsigned long)> >&)ss, std::allocator >*)+0x43) [0x7fc1029578b3] 9: (void decode_noshare(std::map, mempool::mds_co::pool_allocator >, ceph::buffer::v15_2_0::ptr, std::less, mempool::mds_co::pool_allocator > >, mempool::mds_co::pool_allocator, mempool::mds_co::pool_allocator > const, ceph::buffer::v15_2_0::ptr> > >&, ceph::buffer::v15_2_0::list::iterator_impl&)��AٯU0�AٯU�AٯUP�AٯU��AٯU�گU�U��SٯU��SٯU̎��}����3!�������5 ���AٯU�AٯU�GٯU��SٯU`�GٯU�GٯU@P�������� U��CٯUdev_debug_aio�SٯUrue+0x4a5) [0x55afd6d1a7c5] 10: (InodeStoreBase::decode_xattrs(ceph::buffer::v15_2_0::list::iterator_impl&)ss, std::allocator >*)+0x81) [0x55afd6ebb5d1] 11: (InodeStoreBase::decode_bare(ceph::buffer::v15_2_0::list::iterator_impl&, ceph::buffer::v15_2_0::list&, unsigned char))+0x1e1) [0x55afd6ebe6f1] 12: (JournalTool::recover_dentries(EMetaBlob const&, bool, std::set, std::allocator >*)+0x3804) [0x55afd6cb1984] 13: (JournalTool::main_event(std::vector >&)ss, std::allocator >*)+0xdf1) [0x55afd6cb4011] 14: (JournalTool::main(std::vector >&)> >&)ss, std::allocator >*)+0x107c) [0x55afd6c96abc] 15: main() 16: __libc_start_main() 17: _start() 1 jobs: ['5448881'] suites: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/journal-repair}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/quota/quota.sh) on smithi085 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fe900b354658252a2233b17cacef96f5e5d1fe30 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' 2 jobs: ['5448738', '5448852'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Test failure: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) ceph version 16.0.0-5770-gfe900b35 (fe900b354658252a2233b17cacef96f5e5d1fe30) pacific (dev) 1: () 2: gsignal() 3: abort() 4: () 5: () 6: () 7: () 8: (ceph::buffer::v15_2_0::list::iterator_impl::copy(unsigned int, char*) >&)ss, std::allocator >*)+0xb3) [0x7f4bfd8f3c73] 9: (old_inode_t::decode(ceph::buffer::v15_2_0::list::iterator_impl&)ocator >*)+0x5c) [0x55787416d93c] 10: (InodeStoreBase::decode_old_inodes(ceph::buffer::v15_2_0::list::iterator_impl&)rator_impl&)ocator >*)+0x18b) [0x55787430d2db] 11: (InodeStoreBase::decode_bare(ceph::buffer::v15_2_0::list::iterator_impl&, ceph::buffer::v15_2_0::list&, unsigned char))+0x105) [0x55787430d685] 12: (JournalTool::recover_dentries(EMetaBlob const&, bool, std::set, std::allocator >*)+0x2e37) [0x5578741003e7] 13: (JournalTool::main_event(std::vector >&)ss, std::allocator >*)+0xbef) [0x55787410502f] 14: (JournalTool::main(std::vector >&)> >&)ss, std::allocator >*)+0xf63) [0x5578740e8743] 15: main() 16: __libc_start_main() 17: _start() 1 jobs: ['5448767'] suites: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/journal-repair}', 'whitelist_health', 'whitelist_wrongly_marked_down}']