2024-03-14T06:49:16.346 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-14T06:49:16.347 INFO:teuthology.results:seed: '769'
2024-03-14T07:01:37.302 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-14T07:01:37.306 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-14T09:30:48.738 INFO:teuthology.results:Tests finished! gathering results...
2024-03-14T09:30:49.309 INFO:scrape:Found 12 jobs
2024-03-14T09:30:49.314 INFO:scrape:Found 1 distinct failure reasons
2024-03-14T09:30:49.314 INFO:scrape:Failure: "1710401308.8935118 mon.a (mon.0) 519 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
2024-03-14T09:30:49.314 INFO:scrape:12 jobs: ['7599890', '7599899', '7599896', '7599891', '7599892', '7599895', '7599893', '7599897', '7599898', '7599894', '7599900', '7599889']
2024-03-14T09:30:49.314 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/snap-schedule}']
2024-03-14T09:30:49.314 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'tasks/snap-schedule}']
2024-03-14T09:30:49.314 INFO:scrape:
2024-03-14T09:30:49.315 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/
failed:  12
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7599889]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:06:38
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599889/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599889/

    "1710400364.6945422 mon.a (mon.0) 370 : cluster [WRN] Health check failed:
    Reduced data availability: 3 pgs peering (PG_AVAILABILITY)" in cluster log

[7599890]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:12:25
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599890/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599890/

    "1710401308.8935118 mon.a (mon.0) 519 : cluster [WRN] Health check failed:
    Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log

[7599891]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:11:50
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599891/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599891/

    "1710401173.6942863 mon.a (mon.0) 669 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7599892]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:08:40
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599892/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599892/

    "1710401597.6734147 mon.a (mon.0) 1237 : cluster [WRN] Health check failed:
    Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log

[7599893]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:13:45
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599893/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599893/

    "1710400826.4667711 mon.a (mon.0) 476 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7599894]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:15:11
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599894/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599894/

    "1710401364.1776924 mon.a (mon.0) 604 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7599895]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:15:38
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599895/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599895/

    "1710401424.1108415 mon.a (mon.0) 457 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7599896]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:13:22
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599896/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599896/

    "1710401824.0696251 mon.a (mon.0) 1152 : cluster [WRN] Health check failed:
    Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log

[7599897]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:10:54
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599897/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599897/

    "1710401957.5784383 mon.a (mon.0) 1283 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7599898]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:10:12
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599898/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599898/

    "1710401813.561812 mon.a (mon.0) 918 : cluster [WRN] Health check failed:
    Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log

[7599899]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:16:58
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599899/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599899/

    "1710401440.2019987 mon.a (mon.0) 426 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7599900]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule}
-----------------------------------------------------------------
time:   02:10:55
info:   https://pulpito.ceph.com/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599900/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-14_06:47:01-fs-wip-mchangir-64033-quincy-distro-default-smithi/7599900/

    "1710400953.2471943 mon.a (mon.0) 470 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log