Found 160 jobs
Found 12 distinct failure reasons
Failure: SSH connection to smithi036 was lost: 'sudo PATH=/usr/sbin:$PATH lsof ; ps auxf'
23 jobs: ['5408001', '5408017', '5408082', '5408065', '5408011', '5408018', '5408110', '5408059', '5407994', '5408122', '5408106', '5408058', '5408146', '5408089', '5408127', '5408030', '5407999', '5408075', '5408139', '5408047', '5407995', '5408098', '5408111']
suites intersection: ['begin', 'conf/{client', 'fuse-default-perm-no}', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['begin', 'clusters/3-mds', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_norstats}', 'tasks/cfuse_workunit_suites_blogbench}', 'tasks/cfuse_workunit_suites_ffsb}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_fsx}', 'tasks/cfuse_workunit_suites_pjd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Dead: b'2020-09-06T00:06:11.325 DEBUG:teuthology.exit:Finished running handlers'
14 jobs: ['5408124', '5408144', '5408000', '5408016', '5408140', '5408080', '5408060', '5408012', '5408048', '5408096', '5408032', '5408112', '5408064', '5408076']
suites intersection: ['begin', 'conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{fuse-default-perm-no', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['begin', 'ceph-thrash/mds', 'ceph-thrash/mon', 'clusters/3-mds-2-standby', 'clusters/9-mds-3-standby', 'conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multimds/thrash/{0-supported-random-distro$/{centos_8}', 'multimds/thrash/{0-supported-random-distro$/{rhel_8}', 'multimds/thrash/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: "2020-09-05T09:45:14.784878+0000 mon.a (mon.0) 194 : cluster [WRN] Replacing daemon mds.l as rank 0 with standby daemon mds.d" in cluster log
3 jobs: ['5407992', '5408092', '5408141']
suites intersection: ['begin', 'conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['begin', 'ceph-thrash/mds', 'ceph-thrash/mon', 'clusters/3-mds-2-standby', 'clusters/9-mds', 'clusters/9-mds-3-standby', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'multimds/thrash/{0-supported-random-distro$/{rhel_8}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{fuse-default-perm-no', 'q_check_counter/check_counter', 'rhel_8}', 'tasks/cfuse_workunit_suites_dbench}', 'tasks/cfuse_workunit_suites_pjd}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Unable to open channel.
2 jobs: ['5408023', '5408034']
suites intersection: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_kernel_untar_build}', 'tasks/cfuse_workunit_suites_pjd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Test failure: test_ephemeral_pin_dist_conf_off (tasks.cephfs.test_exports.TestEphemeralPins)
5 jobs: ['5408051', '5408099', '5407987', '5408035', '5408115']
suites intersection: ['begin', 'conf/{client', 'fuse-default-perm-no}', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['begin', 'clusters/3-mds', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots)
6 jobs: ['5407990', '5408134', '5408006', '5408054', '5408118', '5408070']
suites intersection: ['begin', 'conf/{client', 'fuse-default-perm-no}', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'tasks/cephfs_test_snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['begin', 'clusters/3-mds', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'tasks/cephfs_test_snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: psutil.NoSuchProcess process no longer exists (pid=5798)
1 jobs: ['5408128']
suites: ['begin', 'ceph-thrash/mon', 'clusters/9-mds-3-standby', 'conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multimds/thrash/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_fsstress}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Command failed (workunit test suites/fsstress.sh) on smithi186 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=db538d78d92e1b8dff340c26a3928d68db1758f8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
1 jobs: ['5408028']
suites: ['begin', 'ceph-thrash/mds', 'clusters/3-mds-2-standby', 'conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multimds/thrash/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_fsstress}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Command failed on smithi002 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel'
2 jobs: ['5408063', '5407996']
suites intersection: ['begin', 'conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'tasks/cfuse_workunit_suites_fsstress}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['begin', 'ceph-thrash/mds', 'clusters/3-mds', 'clusters/3-mds-2-standby', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'multimds/thrash/{0-supported-random-distro$/{rhel_8}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{fuse-default-perm-no', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_suites_fsstress}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: 
1 jobs: ['5408042']
suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_norstats}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: "2020-09-05T13:07:11.567157+0000 mds.b (mds.0) 1 : cluster [WRN] client.4555 isn't responding to mclientcaps(revoke), ino 0x20000000268 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 60.953742 seconds ago" in cluster log
1 jobs: ['5408142']
suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'mds', 'mon', 'mount/fuse', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'tasks/cfuse_workunit_suites_ffsb}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: [Errno 104] Connection reset by peer
1 jobs: ['5408108']
suites: ['begin', 'ceph-thrash/mds', 'clusters/3-mds-2-standby', 'conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multimds/thrash/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_pjd}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']