Found 192 jobs Found 9 distinct failure reasons Failure: '2021-08-05T12:16:37.531671+00:00 smithi006 kernel: [ 662.085139] WARNING: CPU: 7 PID: 27513 at fs/ceph/mds_client.c:4495 check_session_state+0x36/0x60 [ceph] ' in syslog 81 jobs suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides', 'overrides/{no_client_pidfile', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8.stream}', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/acls}', 'tasks/admin}', 'tasks/alternate-pool}', 'tasks/auto-repair}', 'tasks/backtrace}', 'tasks/cap-flush}', 'tasks/client-readahead}', 'tasks/client-recovery}', 'tasks/damage}', 'tasks/exports}', 'tasks/forward-scrub}', 'tasks/fragment}', 'tasks/journal-repair}', 'tasks/mds-flush}', 'tasks/mds-full}', 'tasks/multimds_misc}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/scrub}', 'tasks/sessionmap}', 'tasks/snap-schedule}', 'tasks/strays}', 'tasks/test_journal_migration}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_ops_throttle (tasks.cephfs.test_strays.TestStrays) 1 jobs: ['6312454'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/strays}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: No module named 'tasks.ceph' 11 jobs: ['6312354', '6312375', '6312372', '6312341', '6312353', '6312322', '6312351', '6312373', '6312352', '6312338', '6312374'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin', 'mds', 'mon', 'osd}', 'overrides', 'overrides/{no_client_pidfile', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8.stream}', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/admin}', 'tasks/alternate-pool}', 'tasks/mds-flush}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/snap-schedule}', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2021-08-05T11:53:46.239081+0000 mds.c (mds.1) 1 : cluster [ERR] error reading sessionmap 'mds1_sessionmap' -2 ((2) No such file or directory)" in cluster log 2 jobs: ['6312318', '6312414'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin', 'mds', 'mon', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{no_client_pidfile', 'tasks/journal-repair}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs:functional/{begin', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/journal-repair}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2021-08-05T11:50:28.701746+0000 mon.a (mon.0) 664 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log 7 jobs: ['6312308', '6312447', '6312427', '6312332', '6312331', '6312448', '6312404'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin', 'mds', 'mon', 'osd}', 'overrides', 'overrides/{no_client_pidfile', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8.stream}', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/data-scan}', 'tasks/metrics}', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed on smithi106 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-6694-g2b62472f-1focal ceph-mds=17.0.0-6694-g2b62472f-1focal ceph-common=17.0.0-6694-g2b62472f-1focal ceph-fuse=17.0.0-6694-g2b62472f-1focal ceph-test=17.0.0-6694-g2b62472f-1focal radosgw=17.0.0-6694-g2b62472f-1focal python-ceph=17.0.0-6694-g2b62472f-1focal libcephfs1=17.0.0-6694-g2b62472f-1focal libcephfs-java=17.0.0-6694-g2b62472f-1focal libcephfs-jni=17.0.0-6694-g2b62472f-1focal librados2=17.0.0-6694-g2b62472f-1focal librbd1=17.0.0-6694-g2b62472f-1focal rbd-fuse=17.0.0-6694-g2b62472f-1focal python3-cephfs=17.0.0-6694-g2b62472f-1focal cephfs-shell=17.0.0-6694-g2b62472f-1focal cephfs-top=17.0.0-6694-g2b62472f-1focal cephfs-mirror=17.0.0-6694-g2b62472f-1focal' 1 jobs: ['6312349'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/sessionmap}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: '2021-08-05T12:10:06.536072+00:00 smithi022 kernel: WARNING: CPU: 1 PID: 40483 at fs/ceph/mds_client.c:4456 check_session_state+0x4e/0x60 [ceph] ' in syslog ceph version 17.0.0-6694-g2b62472f (2b62472f690e3a9b00c994d944ad6ee83b335020) quincy (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x7ff064586b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x2c1) [0x7ff065723c9a] 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff065723d62] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x46a) [0x55892e53554c] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x1237) [0x55892e229e21] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x313) [0x55892e23e6c7] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x60) [0x55892e23f32c] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x29a) [0x55892e21a62c] 11: (DispatchQueue::entry()+0x216a) [0x7ff06589bc82] 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff06594e33d] 13: (Thread::entry_wrapper()+0x43) [0x7ff0656f6eb1] 14: (Thread::_entry_func(void*)+0xd) [0x7ff0656f6ecd] 15: /lib64/libpthread.so.0(+0x814a) [0x7ff06457c14a] 16: clone() 2 jobs: ['6312355', '6312356'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8.stream}', 'fs:functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8.stream}', 'fs:functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable) 1 jobs: ['6312336'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/openfiletable}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed on smithi111 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 1 jobs: ['6312333'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{no_client_pidfile', 'tasks/multimds_misc}', 'whitelist_health', 'whitelist_wrongly_marked_down}']