Found 260 jobs
Found 13 distinct failure reasons
Failure: "2021-04-24T21:43:08.234625+0000 mds.b (mds.0) 14 : cluster [WRN] Scrub error on inode 0x100000002cd (/client.0/tmp/tmp) see mds.b log and `damage ls` output for details" in cluster log
14 jobs: ['6067445', '6067423', '6067386', '6067375', '6067401', '6067412', '6067500', '6067454', '6067301', '6067428', '6067514', '6067286', '6067333', '6067338']
suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{begin', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}', 'workunit/suites/ffsb}}', 'workunit/suites/fsstress}}', 'workunit/suites/pjd}}']

Failure: Test failure: test_delete_object (tasks.cephfs.test_volume_client.TestVolumeClient)
2 jobs: ['6067513', '6067329']
suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin', 'mds', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/volume-client}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/volume-client}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage)
1 jobs: ['6067443']
suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Timeout 3h running clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh
2 jobs: ['6067495', '6067368']
suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}']
suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}']

Crash: Command failed on smithi018 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'
ceph version 16.2.1-107-g80c1bb7e (80c1bb7e30aa853a28409be8101c5a37d1e35fd3) pacific (stable)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f29f29623c0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x7f29f2e7eea2]
 5: /usr/lib/ceph/libceph-common.so.2(+0x26404f) [0x7f29f2e7f04f]
 6: (Monitor::~Monitor()+0x144b) [0x557f1132be0b]
 7: (Monitor::~Monitor()+0xd) [0x557f1132be8d]
 8: main()
 9: __libc_start_main()
 10: _start()
1 jobs: ['6067459']
suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}}']

Crash: Command failed on smithi019 with status 1: "sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/client.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' --id 0 /home/ubuntu/cephtest/mnt.0"
ceph version 16.2.1-107-g80c1bb7e (80c1bb7e30aa853a28409be8101c5a37d1e35fd3) pacific (stable)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f83cd871980]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x5648ce6009ce]
 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x5648ce600b58]
 6: ceph-osd(+0xb45dff) [0x5648ce710dff]
 7: (OSDService::~OSDService()+0xd9) [0x5648ce7164b9]
 8: (OSD::~OSD()+0x217) [0x5648ce6c2387]
 9: (OSD::~OSD()+0x9) [0x5648ce6c29e9]
 10: main()
 11: __libc_start_main()
 12: _start()
1 jobs: ['6067366']
suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'distro~HEAD/{ubuntu_latest}', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag_enable', 'overrides}}', 'ranks/5', 'session_timeout', 'tasks/dbench', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Command failed on smithi036 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'
2 jobs: ['6067451', '6067314']
suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['1-client', '2-upgrade', '3-verify}', 'clusters/1-mds-2-client-micro', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro~HEAD/{ubuntu_latest}', 'fs/upgrade/volumes/import-legacy/{bluestore-bitmap', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'ranks/3', 'session_timeout', 'tasks/dbench', 'tasks/{0-nautilus', 'ubuntu_18.04}', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)
2 jobs: ['6067289', '6067487']
suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb'
1 jobs: ['6067268']
suites: ['clusters/small-cluster', 'conf/{client', 'fs/bugs/client_trim_caps/{begin', 'mds', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/trim-i22073}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Test failure: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules)
1 jobs: ['6067342']
suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'}
1 jobs: ['6067312']
suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}}']

Failure: Command failed (workunit test suites/pjd.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=80c1bb7e30aa853a28409be8101c5a37d1e35fd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
1 jobs: ['6067334']
suites: ['2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Command failed (workunit test fs/quota/quota.sh) on smithi139 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=80c1bb7e30aa853a28409be8101c5a37d1e35fd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh'
1 jobs: ['6067377']
suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/workunit/quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}']