Found 154 jobs Found 10 distinct failure reasons Failure: 'NoneType' object has no attribute 'mon_manager' 4 jobs: ['5896021', '5895891', '5895849', '5896045'] suites intersection: ['1-client', '2-upgrade', 'clusters/1-mds-2-client-micro', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-nautilus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-client', '2-upgrade', '3-client-upgrade', '3-compat_client/pacific}}', '4-compat_client', '5-client-sanity}}', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-nautilus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Test failure: test_snapclient_cache (tasks.cephfs.test_snapshots.TestSnapshots), test_snapclient_cache (tasks.cephfs.test_snapshots.TestSnapshots) ceph version 16.1.0-314-g8b9e7177 (8b9e71775081fff7b1c1403e2c8bf67268ef66ba) pacific (rc) 1: /lib64/libpthread.so.0(+0x12b20) [0x7fcd7bdb9b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fcd7d00cb2b] 5: /usr/lib64/ceph/libceph-common.so.2(+0x275cf4) [0x7fcd7d00ccf4] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x128b) [0x55b81678fd9b] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe1c) [0x55b8164d1c7c] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x7bb) [0x55b8164d408b] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x55) [0x55b8164d4685] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x128) [0x55b8164c4648] 11: (DispatchQueue::entry()+0x126a) [0x7fcd7d24479a] 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcd7d2f20a1] 13: /lib64/libpthread.so.0(+0x814a) [0x7fcd7bdaf14a] 14: clone() 1 jobs: ['5895919'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test suites/pjd.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b9e71775081fff7b1c1403e2c8bf67268ef66ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 1 jobs: ['5896015'] suites: ['2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) 2 jobs: ['5895847', '5896068'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2021-02-20T00:00:39.198826+0000 mds.c (mds.0) 14 : cluster [WRN] Scrub error on inode 0x10000000b97 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/mce-test) see mds.c log and `damage ls` output for details" in cluster log 3 jobs: ['5896025', '5895905', '5895957'] suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'mds', 'mon', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}', 'workunit/suites/fsstress}}'] Failure: Command failed (workunit test libcephfs/test.sh) on smithi150 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b9e71775081fff7b1c1403e2c8bf67268ef66ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 1 jobs: ['5896028'] suites: ['clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin', 'mds', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/libcephfs}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 1 jobs: ['5896024'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: "2021-02-20T00:25:27.244345+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log ceph version 16.1.0-314-g8b9e7177 (8b9e71775081fff7b1c1403e2c8bf67268ef66ba) pacific (rc) 1: /lib64/libpthread.so.0(+0x12dc0) [0x7fc33a5cedc0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9006b) [0x7fc339bf106b] 5: /lib64/libstdc++.so.6(+0x9650c) [0x7fc339bf750c] 6: /lib64/libstdc++.so.6(+0x96567) [0x7fc339bf7567] 7: /lib64/libstdc++.so.6(+0x967c8) [0x7fc339bf77c8] 8: ceph-osd(+0x587fa6) [0x561975f87fa6] 9: (ceph::buffer::v15_2_0::create_aligned(unsigned int, unsigned int)+0x26) [0x56197689bdc6] 10: (ceph::buffer::v15_2_0::create_small_page_aligned(unsigned int)+0x55) [0x56197689c815] 11: (ProtocolV1::read_message_data_prepare()+0x368) [0x561976a3e468] 12: (ProtocolV1::read_message_middle()+0x120) [0x561976a3e620] 13: (ProtocolV1::handle_message_front(char*, int)+0x2f4) [0x561976a3ef84] 14: ceph-osd(+0x103854d) [0x561976a3854d] 15: (AsyncConnection::process()+0x789) [0x561976a355a9] 16: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xcb7) [0x5619768853f7] 17: ceph-osd(+0xe88f1c) [0x561976888f1c] 18: /lib64/libstdc++.so.6(+0xc2b73) [0x7fc339c23b73] 19: /lib64/libpthread.so.0(+0x82de) [0x7fc33a5c42de] 20: clone() 1 jobs: ['5896060'] suites: ['2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/quota/quota.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b9e71775081fff7b1c1403e2c8bf67268ef66ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' 1 jobs: ['5895967'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/workunit/quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 1 jobs: ['5896041'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}']