Found 259 jobs Found 15 distinct failure reasons Dead: b'2021-04-16T02:54:54.699 DEBUG:teuthology.exit:Finished running handlers' ceph version 17.0.0-3238-g3b383aa3 (3b383aa3978386fbc36cbca292c135c6dbedcc66) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f8eb759e3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x56307ea2c207] 5: (KernelDevice::_aio_thread()+0x11a5) [0x56307f658fb5] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x56307f663921] 7: (Thread::_entry_func(void*)+0xd) [0x56307f2350ed] 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f8eb7592609] 9: clone() 1 jobs: ['6047654'] suites: ['clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/snaps/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/workunit/snaps}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_subvolume_evict_client (tasks.cephfs.test_volumes.TestSubvolumes), test_subvolume_evict_client (tasks.cephfs.test_volumes.TestSubvolumes) 1 jobs: ['6047701'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) 1 jobs: ['6047578'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2021-04-15T19:01:01.896215+0000 osd.4 (osd.4) 14 : cluster [ERR] 5.0s0 deep-scrub : stat mismatch, got 0/21 objects, 0/0 clones, 0/21 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/59615 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log 2 jobs: ['6047716', '6047582'] suites intersection: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/fs/snaps}}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: SSH connection to smithi077 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' ceph version 17.0.0-3238-g3b383aa3 (3b383aa3978386fbc36cbca292c135c6dbedcc66) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f261490d3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f2614bbaf6c] 5: /usr/lib/ceph/libceph-common.so.2(+0x26510a) [0x7f2614bbb10a] 6: (Thread::join(void**)+0x113) [0x7f2614ca3c93] 7: (Finisher::stop()+0x7e) [0x7f2614c681ae] 8: (MDSRankDispatcher::shutdown()+0x3e8) [0x55e1f62e6a08] 9: (MDSDaemon::suicide()+0x310) [0x55e1f62d4ce0] 10: (Context::complete(int)+0xd) [0x55e1f62dc5bd] 11: (MDSRank::suicide()+0x1c) [0x55e1f62e288c] 12: (MDSRank::boot_start(MDSRank::BootStep, int)+0xc86) [0x55e1f62f90d6] 13: (MDSRank::_standby_replay_restart_finish(int, unsigned long)+0x178) [0x55e1f62f9eb8] 14: (MDSContext::complete(int)+0x56) [0x55e1f6641cc6] 15: (MDSIOContextBase::complete(int)+0x243) [0x55e1f6642133] 16: (Finisher::finisher_thread_entry()+0x19d) [0x7f2614c68c9d] 17: (Thread::_entry_func(void*)+0xd) [0x7f2614ca394d] 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f2614901609] 19: clone() 1 jobs: ['6047611'] suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}}'] Failure: "2021-04-15T15:37:49.801012+0000 mon.a (mon.0) 3156 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.c" in cluster log 4 jobs: ['6047526', '6047521', '6047603', '6047525'] suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/volumes/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/5', 'rhel_8}', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/admin}', 'tasks/volumes/{overrides', 'tasks/{0-check-counter', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/kernel_untar_build}}'] Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi036 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b383aa3978386fbc36cbca292c135c6dbedcc66 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2 jobs: ['6047630', '6047690'] suites intersection: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'osd}', 'overrides/{frag_enable', 'ranks/1', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: failed to install new kernel version within timeout 1 jobs: ['6047759'] suites: ['2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Dead: b'2021-04-16T02:54:54.700 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['6047657'] suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/dbench}}'] Failure: "2021-04-15T15:38:08.555469+0000 mds.f (mds.2) 29 : cluster [WRN] Scrub error on inode 0x100000080ea (/client.0/tmp/payload.1/multiple_rsync_payload.150501/firewalld/zones) see mds.f log and `damage ls` output for details" in cluster log 1 jobs: ['6047523'] suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'ranks/3', 'rhel_8}', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}'] Failure: Test failure: test_rebuild_backtraceless (tasks.cephfs.test_data_scan.TestDataScan) 1 jobs: ['6047586'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/data-scan}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: SSH connection to smithi023 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 1 jobs: ['6047542'] suites: ['2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Found coredumps on ubuntu@smithi052.front.sepia.ceph.com ceph version 17.0.0-3238-g3b383aa3 (3b383aa3978386fbc36cbca292c135c6dbedcc66) quincy (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f16f4579b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a0) [0x55e4e1ebd600] 5: ceph-osd(+0x5b17ba) [0x55e4e1ebd7ba] 6: (SharedLRU::~SharedLRU()+0x353) [0x55e4e2047413] 7: (OSDService::~OSDService()+0xd9) [0x55e4e204d199] 8: (OSD::~OSD()+0x22c) [0x55e4e1ffba6c] 9: (OSD::~OSD()+0xd) [0x55e4e1ffc02d] 10: main() 11: __libc_start_main() 12: _start() 1 jobs: ['6047728'] suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/5', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) 1 jobs: ['6047623'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2021-04-15T18:06:40.267938+0000 mds.a (mds.2) 6 : cluster [WRN] Scrub error on inode 0x100000010eb (/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-17) see mds.a log and `damage ls` output for details" in cluster log 1 jobs: ['6047694'] suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}}']