Found 146 jobs Found 4 distinct failure reasons Failure: The WatchDog barked due to Daemon ceph.osd.3 has failed 2 jobs: ['68231', '68291'] suites intersection: ['0-size-min-size-overrides/3-size-2-min-size', '2-recovery-overrides/{default}', 'clusters/{crimson-fixed-2}', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'thrashers/{pggrow}', 'thrashosds-health'] suites union: ['0-size-min-size-overrides/3-size-2-min-size', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'clusters/{crimson-fixed-2}', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/seastore/segmented$/{crimson_seastore_segmented_2q}', 'objectstore/seastore/segmented$/{crimson_seastore_segmented}', 'thrashers/{pggrow}', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'workloads/snaps-few-objects-balanced}'] Failure: Command failed (workunit test rados/test.sh) on trial036 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 1 jobs: ['68276'] suites: ['clusters/crimson-fixed-2', 'crimson-rados/basic/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/crimson_bluestore', 'tasks/rados_api_tests}'] Failure: failed to complete snap trimming before timeout 1 jobs: ['68171'] suites: ['0-size-min-size-overrides/3-size-2-min-size', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'clusters/{crimson-fixed-2}', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/seastore/rbm$/{crimson_seastore_rbm}', 'thrashers/{pggrow}', 'thrashosds-health', 'workloads/pool-snaps-few-objects}'] Failure: reached maximum tries (500) after waiting for 3000 seconds 2 jobs: ['68292', '68193'] suites intersection: ['0-size-min-size-overrides/3-size-2-min-size', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'clusters/{crimson-fixed-2}', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'thrashosds-health'] suites union: ['0-size-min-size-overrides/3-size-2-min-size', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'clusters/{crimson-fixed-2}', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/crimson_bluestore', 'objectstore/seastore/segmented$/{crimson_seastore_segmented}', 'thrashers/{default}', 'thrashers/{pggrow}', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/radosbench-high-concurrency}']