Found 141 jobs Not-understood crash indication Binary file /home/teuthworker/archive/teuthology-2020-09-06_03:15:03-fs-master-distro-basic-smithi/5410296/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/teuthology-2020-09-06_03:15:03-fs-master-distro-basic-smithi/5410296/teuthology.log matches Not-understood crash indication Found 5 distinct failure reasons Timeout 120 running ceph --cluster ceph --log-early osd dump --format=json 5 jobs: ['5410296', '5410314', '5410239', '5410254', '5410356'] suites intersection: ['1-client', '2-upgrade', 'clusters/1-mds-2-client-micro', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-nautilus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-client', '2-upgrade', '3-compat_client/no}}', '3-compat_client/pacific}}', '3-verify}', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/volumes/import-legacy/{bluestore-bitmap', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-nautilus', 'ubuntu_18.04}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Test failure: test_standby_replay_singleton_fail_multimds (tasks.cephfs.test_failover.TestStandbyReplay) ceph version 16.0.0-5167-g0f0addb4d7 (0f0addb4d75645359bb3ecb3b09183da3c2d495e) pacific (dev) 1: (()+0x12dc0) [0x7fa23e0ecdc0] 2: (gsignal()+0x10f) [0x7fa23cbb08df] 3: (abort()+0x127) [0x7fa23cb9acf5] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fa23f4f98e1] 5: (()+0x276aaa) [0x7fa23f4f9aaa] 6: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x1bf) [0x55a8fffc464f] 7: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x128) [0x55a8fffb25b8] 8: (DispatchQueue::entry()+0x126a) [0x7fa23f72135a] 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa23f7cfec1] 10: (()+0x82de) [0x7fa23e0e22de] 11: (clone()+0x43) [0x7fa23cc75133] 1 jobs: ['5410347'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/failover}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed on smithi140 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' 1 jobs: ['5410275'] suites: ['clusters/fixed-2-ucephfs', 'conf/{client', 'fs/basic_workload/{begin', 'inline/no', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_kernel_untar_build}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/quota/quota.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0f0addb4d75645359bb3ecb3b09183da3c2d495e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' 2 jobs: ['5410327', '5410257'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'tasks/cfuse_workunit_quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) 2 jobs: ['5410300', '5410368'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'tasks/test_journal_migration}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/test_journal_migration}', 'whitelist_health', 'whitelist_wrongly_marked_down}']