Found 314 jobs Found 9 distinct failure reasons Failure: "2020-11-25T03:39:55.686437+0000 mon.a (mon.0) 1793 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log 2 jobs: ['5654380', '5654339'] suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'fs/mixed-clients/{begin', 'k-testing}', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'rhel_8}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/strays}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2 jobs: ['5654150', '5654307'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'rhel_8}', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_subvolumegroup_pin_distributed (tasks.cephfs.test_volumes.TestSubvolumeGroups) 1 jobs: ['5654138'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Command failed on smithi163 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' ceph version 16.0.0-7496-g5491e7e6 (5491e7e6c4f333ab8b271abd4c5956530b8bfaa2) pacific (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9ee79d9b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x555ce3e70427] 5: (KernelDevice::_aio_thread()+0x1254) [0x555ce496db44] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x555ce4978881] 7: /lib64/libpthread.so.0(+0x814a) [0x7f9ee79cf14a] 8: clone() 1 jobs: ['5654329'] suites: ['clusters/1a2s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'tasks/workunit/suites/iogen}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Dead: b'2020-11-25T10:41:31.640 DEBUG:teuthology.exit:Finished running handlers' 3 jobs: ['5654119', '5654350', '5654267'] suites intersection: ['2-workunit/cfuse_workunit_suites_fsstress}}', 'conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1-mds-1-client-coloc', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: 'NoneType' object has no attribute 'get' 2 jobs: ['5654318', '5654184'] suites intersection: ['1-client', '2-upgrade', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-nautilus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-client', '2-upgrade', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-nautilus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) 1 jobs: ['5654335'] suites: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test suites/ffsb.sh) on smithi089 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5491e7e6c4f333ab8b271abd4c5956530b8bfaa2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 1 jobs: ['5654255'] suites: ['2-workunit/ffsb}}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_subvolume_retain_snapshot_invalid_recreate (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) 1 jobs: ['5654263'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/snapshot}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']