Found 112 jobs Found 3 distinct failure reasons Crash: "2021-02-15T10:31:33.158123+0000 mon.a (mon.0) 565 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log ceph version 17.0.0-775-gdeecabd7 (deecabd7c371b592cc0e4400aa9bc46814931e51) quincy (dev) 1: /lib64/libpthread.so.0(+0x12dc0) [0x7f0ab2f5adc0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9006b) [0x7f0ab257d06b] 5: /lib64/libstdc++.so.6(+0x9650c) [0x7f0ab258350c] 6: /lib64/libstdc++.so.6(+0x95529) [0x7f0ab2582529] 7: __gxx_personality_v0() 8: /lib64/libgcc_s.so.1(+0x10b13) [0x7f0ab1f63b13] 9: _Unwind_Resume() 10: ceph-osd(+0x56712e) [0x55a18ad0c12e] 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a18b4994e4] 12: (Thread::_entry_func(void*)+0xd) [0x55a18b48834d] 13: /lib64/libpthread.so.0(+0x82de) [0x7f0ab2f502de] 14: clone() 1 jobs: ['5883907'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] Failure: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi086 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=deecabd7c371b592cc0e4400aa9bc46814931e51 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 9 jobs: ['5883857', '5883951', '5883923', '5883870', '5883873', '5883934', '5883959', '5883911', '5883859'] suites intersection: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'openstack}', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install'] suites union: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}'] Failure: "2021-02-15T10:24:22.326756+0000 mon.a (mon.0) 201 : cluster [WRN] Health check failed: Degraded data redundancy: 3/3678 objects degraded (0.082%), 1 pg degraded (PG_DEGRADED)" in cluster log 2 jobs: ['5883914', '5883851'] suites intersection: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}'] suites union: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-journal-stress-workunit}']