Found 40 jobs
Found 8 distinct failure reasons
Failure: Extra data: line 2 column 557 (char 557)
2 jobs: ['6806771', '6806752']
suites intersection: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Dead: b'2022-04-26T16:49:23.832 DEBUG:teuthology.exit:Finished running handlers'
4 jobs: ['6806750', '6806755', '6806778', '6806741']
suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['1-cephadm', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{pg-warn', 'ranks/3', 'roles', 'scrub/yes', 'session_timeout', 'standby-replay', 'syntax', 'tasks/{0-check-counter', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}', 'wsync/{no}}']

Failure: "2022-04-26T11:17:10.983166+0000 mon.a (mon.0) 1272 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
1 jobs: ['6806762']
suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: reached maximum tries (90) after waiting for 540 seconds
4 jobs: ['6806746', '6806749', '6806775', '6806773']
suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{no}}', 'wsync/{yes}}']

Timeout 3h running clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh
1 jobs: ['6806764']
suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: "2022-04-26T11:11:55.833329+0000 mds.d (mds.1) 1 : cluster [WRN] client.4691 isn't responding to mclientcaps(revoke), ino 0x200000003cb pending pAsxLsXsxFscr issued pAsxLsXsxFsxcrwb, sent 300.004295 seconds ago" in cluster log
3 jobs: ['6806776', '6806779', '6806770']
suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Failure: Command crashed: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid da8c282a-c54c-11ec-8c39-001a4aab830c -e sha1=04b6d965e55961a6cb690e35b6f77d0192e1fe45 -- bash -c \'while ceph orch upgrade status | jq \'"\'"\'.in_progress\'"\'"\' | grep true && ! ceph orch upgrade status | jq \'"\'"\'.message\'"\'"\' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done\''
1 jobs: ['6806760']
suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']

Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
1 jobs: ['6806772']
suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']