Found 24 jobs
Found 5 distinct failure reasons
Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=centos%2F7%2Fx86_64&sha1=a5269762c4b7bb3655b3e41168328fce85946603
8 jobs: ['5199941', '5199929', '5199927', '5199943', '5199903', '5199915', '5199913', '5199917']
suites intersection: ['conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}', 'whitelist_health']
suites union: ['begin', 'centos_latest', 'ceph-thrash/default', 'clusters/3-mds', 'clusters/3-mds-2-standby', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'inline/yes', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multimds/basic/{0-supported-random-distro$/{centos_latest}', 'multimds/thrash/{0-supported-random-distro$/{centos_latest}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{fuse-default-perm-no', 'q_check_counter/check_counter', 'tasks/cephfs_test_exports}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_blogbench}', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_fsstress}', 'thrash/{frag_enable', 'thrash_debug}', 'validater/lockdep}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'whitelist_wrongly_marked_down}}']

Failure: "2020-07-05 15:03:08.640081 mon.b (mon.0) 759 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log
1 jobs: ['5199921']
suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{kernel-testing', 'ms-die-on-skipped}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']

Failure: '/home/ubuntu/cephtest/archive/syslog/kern.log:2020-07-05T13:48:27.359403+00:00 smithi132 kernel: [ 1437.528260] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:28678]
' in syslog
1 jobs: ['5199919']
suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_16.04}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/random/{k-testing', 'q_check_counter/check_counter', 'supported$/{ubuntu_latest}}', 'tasks/cfuse_workunit_suites_fsx}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Command failed (workunit test suites/fsstress.sh) on smithi196 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=120f482407ab4070126cb7ffc2f4f1b37517b4a0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
1 jobs: ['5199931']
suites: ['begin', 'ceph-thrash/default', 'clusters/3-mds-2-standby', 'conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multimds/thrash/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{distro/rhel/{k-distro', 'overrides/{fuse-default-perm-no', 'rhel_latest}', 'tasks/cfuse_workunit_suites_fsstress}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: '/home/ubuntu/cephtest/archive/syslog/misc.log:2020-07-05T13:51:24.274119+00:00 smithi189 crond[42958]: (root) INFO (Job execution of per-minute job scheduled for 13:50 delayed into subsequent minute 13:51. Skipping job run.)
' in syslog
1 jobs: ['5199911']
suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_16.04}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/rhel/{k-distro', 'q_check_counter/check_counter', 'rhel_latest}', 'tasks/cfuse_workunit_suites_ffsb}', 'whitelist_health', 'whitelist_wrongly_marked_down}']