Found 69 jobs Found 5 distinct failure reasons Failure: "2023-03-22T10:21:40.243609+0000 mon.a (mon.0) 970 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 3 jobs: ['7211755', '7211742', '7211692'] suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snap-schedule}', 'tasks/strays}', 'tasks/volumes/{overrides', 'test/clone}}'] Failure: Command failed on smithi043 with status 1: 'set -ex\nsudo dd of=/home/ubuntu/cephtest/valgrind.supp' 1 jobs: ['7211710'] suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'fs/permission/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_pjd}'] Failure: "2023-03-22T00:12:26.223211+0000 osd.5 (osd.5) 3 : cluster [WRN] OSD bench result of 226833.495141 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.5. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 1 jobs: ['7211719'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}'] Dead: b'2023-03-22T11:57:45.938 DEBUG:teuthology.exit:Finished running handlers' 2 jobs: ['7211711', '7211726'] suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{frag', 'ranks/5'] suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'scrub/no', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'workunit/suites/fsstress}', 'wsync/{yes}}'] Crash: Test failure: test_parallel_execution (tasks.cephfs.test_data_scan.TestDataScan) ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4cc090bb20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4cc190e4dd] 5: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7f4cc190e649] 6: (MDLog::_recovery_thread(MDSContext*)+0xb56) [0x55ba8b5e2096] 7: (MDLog::RecoveryThread::entry()+0x15) [0x55ba8b296695] 8: /lib64/libpthread.so.0(+0x814a) [0x7f4cc090114a] 9: clone() 1 jobs: ['7211707'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/data-scan}']