Found 134 jobs Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837340/remote/gibba029/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837340/remote/gibba029/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837340/remote/gibba029/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837403/remote/gibba006/log/ceph-mgr.y.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837403/remote/gibba006/log/ceph-mgr.y.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837403/remote/gibba006/log/ceph-mgr.y.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837403/remote/gibba006/log/ceph-mgr.y.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837403/remote/gibba006/log/ceph-mgr.y.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837403/remote/gibba006/log/ceph-mgr.y.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Missing log /home/teuthworker/archive/teuthology-2021-01-29_03:15:04-fs-master-distro-basic-gibba/5837380/remote/gibba031/log/ceph-mgr.x.log.gz Not-understood crash indication Found 13 distinct failure reasons Failure: Command failed (workunit test suites/dbench.sh) on gibba029 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9fea82cacc62bbd484723cdbc8c6729a89d7f23c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 1 jobs: ['5837340'] suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'scrub/no', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/dbench}}'] Crash: Test failure: test_snapclient_cache (tasks.cephfs.test_snapshots.TestSnapshots), test_snapclient_cache (tasks.cephfs.test_snapshots.TestSnapshots) ceph version 17.0.0-208-g9fea82ca (9fea82cacc62bbd484723cdbc8c6729a89d7f23c) pacific (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f991fd4c3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x7f991fff7d8a] 5: /usr/lib/ceph/libceph-common.so.2(+0x262f37) [0x7f991fff7f37] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x14b0) [0x55c97a681b90] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe85) [0x55c97a375d35] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x643) [0x55c97a38cd83] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5e) [0x55c97a38d41e] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x128) [0x55c97a3694e8] 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x460) [0x7f99202400e0] 12: (DispatchQueue::entry()+0x58f) [0x7f992023d97f] 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9920304bf1] 14: (Thread::_entry_func(void*)+0xd) [0x7f99200e4e2d] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f991fd40609] 16: clone() 1 jobs: ['5837376'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_discontinuous_mdsmap (tasks.cephfs.test_failover.TestFailover), test_discontinuous_mdsmap (tasks.cephfs.test_failover.TestFailover) 1 jobs: ['5837437'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/multifs/{begin', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/failover}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: SSH connection to gibba006 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 4 jobs: ['5837403', '5837391', '5837358', '5837453'] suites intersection: ['conf/{client', 'k-testing}', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/suites/pjd}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/mixed-clients/{begin', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'k-testing}', 'kclient-overrides/{distro/testing/{flavor/ubuntu_latest', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'session_timeout', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: SSH connection to gibba032 was lost: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9fea82cacc62bbd484723cdbc8c6729a89d7f23c' 5 jobs: ['5837445', '5837438', '5837394', '5837450', '5837418'] suites intersection: ['conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/cfuse_workunit_suites_pjd}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/snaps/{begin', 'fs/thrash/multifs/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/5', 'scrub/no', 'session_timeout', 'tasks/workunit/quota}', 'tasks/workunit/snaps}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/kernel_untar_build}}', 'workunit/suites/iozone}}'] Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) 1 jobs: ['5837338'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2021-01-31T11:19:35.283042+0000 mds.f (mds.0) 119 : cluster [WRN] Scrub error on inode 0x1000000140a (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces) see mds.f log and `damage ls` output for details" in cluster log 4 jobs: ['5837374', '5837354', '5837442', '5837457'] suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{begin', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/3', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}', 'workunit/suites/ffsb}}', 'workunit/suites/fsstress}}'] Failure: Test failure: test_subvolumegroup_pin_distributed (tasks.cephfs.test_volumes.TestSubvolumeGroups) 1 jobs: ['5837411'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_subvolume_snapshot_clone_different_groups (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 1 jobs: ['5837440'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb' 1 jobs: ['5837343'] suites: ['clusters/small-cluster', 'conf/{client', 'fs/bugs/client_trim_caps/{begin', 'mds', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/trim-i22073}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test suites/fsx.sh) on gibba031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9fea82cacc62bbd484723cdbc8c6729a89d7f23c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 1 jobs: ['5837380'] suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/no', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/fsx}}'] Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 1 jobs: ['5837429'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_ephemeral_random_dist (tasks.cephfs.test_exports.TestEphemeralPins) 1 jobs: ['5837441'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']