Found 39 jobs
Found 9 distinct failure reasons
Failure: timeout expired in wait_until_healthy
9 jobs: ['7641249', '7641239', '7641243', '7641245', '7641230', '7641226', '7641220', '7641223', '7641232']
suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}']
suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04']

Failure: Command failed on smithi148 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.30 deep-scrub'
2 jobs: ['7641246', '7641227']
suites intersection: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus', 'upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap']
suites union: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus', 'upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap']

Failure: "2024-04-05T04:50:00.000279+0000 mon.a (mon.0) 6304 : cluster 4 [ERR] OSD_SCRUB_ERRORS: 8 scrub errors" in cluster log
11 jobs: ['7641231', '7641248', '7641219', '7641225', '7641228', '7641221', '7641217', '7641244', '7641214', '7641242', '7641235']
suites intersection: []
suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04', 'workload/{ec-rados-default']

Failure: Command failed on smithi191 with status 1: 'sudo yum install -y kernel'
1 jobs: ['7641236']
suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream}', 'workload/{ec-rados-default']

Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
2 jobs: ['7641247', '7641224']
suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']
suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.1}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']

Failure: reached maximum tries (51) after waiting for 300 seconds
3 jobs: ['7641241', '7641229', '7641216']
suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']
suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']

Dead: b'2024-04-05T14:34:32.377 DEBUG:teuthology.exit:Finished running handlers'
2 jobs: ['7641238', '7641215']
suites intersection: []
suites union: ['0-roles', '0-start', '1-start', '1-tasks}', '2-first-half-tasks/rbd-import-export', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_8.stream_container_tools_crun}']

Failure: "1712286347.7461145 mon.a (mon.0) 23 : cluster [ERR] Health check failed: 8 osds(s) are not reachable (OSD_UNREACHABLE)" in cluster log
1 jobs: ['7641233']
suites: ['0-start', '1-tasks}', 'upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{ubuntu_22.04}']

Failure: Failed to reconnect to smithi191
1 jobs: ['7641237']
suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.1}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']