2024-02-29T17:43:35.398 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-29T17:43:35.398 INFO:teuthology.results:subset: '111/200'
2024-02-29T17:43:35.398 INFO:teuthology.results:seed: '3408'
2024-02-29T17:45:28.490 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-29T17:45:28.494 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-29T20:12:40.205 INFO:teuthology.results:Tests finished! gathering results...
2024-02-29T20:12:40.596 INFO:scrape:Found 4 jobs
2024-02-29T20:12:45.395 INFO:scrape:Found 3 distinct failure reasons
2024-02-29T20:12:45.395 INFO:scrape:Failure: failed to complete snap trimming before timeout
2024-02-29T20:12:45.395 INFO:scrape:2 jobs: ['7577228', '7577229']
2024-02-29T20:12:45.396 INFO:scrape:suites intersection: []
2024-02-29T20:12:45.396 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/classic', 'mon_election/classic}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:quincy-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade:quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'workload/{ec-rados-default']
2024-02-29T20:12:45.396 INFO:scrape:
2024-02-29T20:12:45.396 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=quincy
2024-02-29T20:12:45.396 INFO:scrape:1 jobs: ['7577226']
2024-02-29T20:12:45.396 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd_api', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/ubuntu_22.04']
2024-02-29T20:12:45.396 INFO:scrape:
2024-02-29T20:12:45.396 INFO:scrape:Failure: "1709229815.794988 mon.a (mon.0) 517 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2024-02-29T20:12:45.396 INFO:scrape:1 jobs: ['7577227']
2024-02-29T20:12:45.396 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/centos_9.stream']
2024-02-29T20:12:45.397 INFO:scrape:
2024-02-29T20:12:45.397 INFO:teuthology.results:Sending results to akupczyk@redhat.com: Test Run: akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7577226]  upgrade:quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   00:07:08
info:   https://pulpito.ceph.com/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/7577226/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/7577226/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e9da99ffd8914e37a798f9883beb726e

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=
    quincy

[7577227]  upgrade:quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
-----------------------------------------------------------------
time:   02:20:07
info:   https://pulpito.ceph.com/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/7577227/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/7577227/

    "1709229815.794988 mon.a (mon.0) 517 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7577228]  upgrade:quincy-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   00:42:53
info:   https://pulpito.ceph.com/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/7577228/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/7577228/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ea79ba0e959a4bb397535a06953f57e4

    failed to complete snap trimming before timeout

[7577229]  upgrade:quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
-----------------------------------------------------------------
time:   00:45:19
info:   https://pulpito.ceph.com/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/7577229/
log:    http://qa-proxy.ceph.com/teuthology/akupczyk-2024-02-29_17:43:25-upgrade:quincy-x-main-distro-default-smithi/7577229/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b9eea00eda1842d9a590e2ac3b6ee3fb

    failed to complete snap trimming before timeout