Found 41 jobs
Found 9 distinct failure reasons
Failure: timeout expired in wait_until_healthy
9 jobs: ['7653395', '7653423', '7653416', '7653398', '7653427', '7653406', '7653409', '7653420', '7653402']
suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}']
suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04']

Failure: "1712911170.7989874 mon.a (mon.0) 522 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
3 jobs: ['7653392', '7653413', '7653410']
suites intersection: ['0-start']
suites union: ['0-start', '1-tasks', '1-tasks}', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream_runc}', 'workload/{ec-rados-default']

Failure: "2024-04-12T08:50:18.574827+0000 mon.a (mon.0) 225 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log
11 jobs: ['7653394', '7653419', '7653404', '7653415', '7653396', '7653400', '7653422', '7653389', '7653426', '7653412', '7653408']
suites intersection: []
suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04', 'workload/{ec-rados-default']

Failure: Command failed on smithi078 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd'
2 jobs: ['7653421', '7653407']
suites intersection: ['1-mount/mount/fuse', '2-clients/fuse-upgrade', 'branch/pacific', 'centos_8.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-install', 'upgrade/cephfs/upgraded_client/{bluestore-bitmap']
suites union: ['1-mount/mount/fuse', '2-clients/fuse-upgrade', '3-workload/stress_tests/blogbench}}', '3-workload/stress_tests/fsstress}}', 'branch/pacific', 'centos_8.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-install', 'upgrade/cephfs/upgraded_client/{bluestore-bitmap']

Dead: b'2024-04-12T20:24:24.882 DEBUG:teuthology.exit:Finished running handlers'
1 jobs: ['7653390']
suites: ['0-start', '1-tasks}', 'upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_8.stream_container_tools_crun}']

Failure: Command failed on smithi047 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.35 deep-scrub'
1 jobs: ['7653403']
suites: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus', 'upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap']

Failure: reached maximum tries (51) after waiting for 300 seconds
3 jobs: ['7653399', '7653414', '7653418']
suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']
suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']

Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
1 jobs: ['7653425']
suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']

Failure: ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8.
1 jobs: ['7653393']
suites: ['1-upgrade}}', 'centos_8.stream', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-pacific', 'upgrade/cephfs/nofs/{bluestore-bitmap']