Found 320 jobs Found 10 distinct failure reasons Failure: "2020-10-09T19:57:01.787755+0000 mds.f (mds.6) 1 : cluster [WRN] client.4744 isn't responding to mclientcaps(revoke), ino 0x20000000262 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 63.847564 seconds ago" in cluster log 2 jobs: ['5507760', '5507696'] suites intersection: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_suites_ffsb}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'mds', 'mon', 'mount/fuse', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_suites_ffsb}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2020-10-09T15:22:15.362652+0000 mds.b (mds.0) 1 : cluster [WRN] client.4532 isn't responding to mclientcaps(revoke), ino 0x1000000074b pending pAsLsXsFr issued pAsLsXsFrw, sent 62.515374 seconds ago" in cluster log 1 jobs: ['5507584'] suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_suites_ffsb}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: 4 jobs: ['5507495', '5507527', '5507559', '5507463'] suites intersection: ['centos_latest', 'clusters/3-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'osd}', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}'] suites union: ['centos_latest', 'clusters/3-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}'] Failure: Command failed (workunit test suites/pjd.sh) on smithi194 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e783bcc4ed94e312ec724570ede3e64902f921b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2 jobs: ['5507716', '5507476'] suites intersection: ['begin', 'conf/{client', 'fuse-default-perm-no}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_suites_pjd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['begin', 'clusters/3-mds', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'rhel_8}', 'tasks/cfuse_workunit_suites_pjd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/misc/dirfrag.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e783bcc4ed94e312ec724570ede3e64902f921b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/misc/dirfrag.sh' 1 jobs: ['5507547'] suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/stock/{k-stock', 'q_check_counter/check_counter', 'rhel_8}', 'tasks/cfuse_workunit_misc}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Timeout 3h running clone.client.1/qa/workunits/suites/blogbench.sh 1 jobs: ['5507757'] suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_suites_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed on smithi008 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 1 jobs: ['5507513'] suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_kernel_untar_build}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed on smithi152 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' 1 jobs: ['5507693'] suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_suites_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Timeout 3h running clone.client.1/qa/workunits/kernel_untar_build.sh 1 jobs: ['5507705'] suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_kernel_untar_build}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Command failed (workunit test suites/fsstress.sh) on smithi132 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e783bcc4ed94e312ec724570ede3e64902f921b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' ceph version 16.0.0-6231-ge783bcc4 (e783bcc4ed94e312ec724570ede3e64902f921b4) pacific (dev) 1: /lib64/libpthread.so.0(+0x12dc0) [0xef10dc0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x52f8be9] 5: (Client::_do_remount(bool)+0x3ce) [0x1a606e] 6: (Context::complete(int)+0xd) [0x21127d] 7: (Finisher::finisher_thread_entry()+0x28d) [0x539bb1d] 8: /lib64/libpthread.so.0(+0x82de) [0xef062de] 9: clone() 1 jobs: ['5507591'] suites: ['centos_latest', 'clusters/3-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']