2024-03-02T17:40:17.873 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-02T17:40:17.873 INFO:teuthology.results:seed: '3534'
2024-03-02T17:43:43.516 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-02T17:43:43.538 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-02T18:21:46.566 INFO:teuthology.results:Tests finished! gathering results...
2024-03-02T18:21:47.036 INFO:scrape:Found 12 jobs
2024-03-02T18:21:56.450 INFO:scrape:Found 2 distinct failure reasons
2024-03-02T18:21:56.450 INFO:scrape:Failure: Test failure: test_quiesce_path_release (tasks.cephfs.test_quiesce.TestQuiesce)
2024-03-02T18:21:56.450 INFO:scrape:4 jobs: ['7579917', '7579926', '7579923', '7579920']
2024-03-02T18:21:56.450 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/quiesce}']
2024-03-02T18:21:56.450 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'tasks/quiesce}']
2024-03-02T18:21:56.451 INFO:scrape:
2024-03-02T18:21:56.451 INFO:scrape:Failure: Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)
2024-03-02T18:21:56.451 INFO:scrape:8 jobs: ['7579918', '7579919', '7579921', '7579922', '7579924', '7579925', '7579916', '7579927']
2024-03-02T18:21:56.451 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/quiesce}']
2024-03-02T18:21:56.451 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'tasks/quiesce}']
2024-03-02T18:21:56.451 INFO:scrape:
2024-03-02T18:21:56.452 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/
failed:  12
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7579916]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quiesce}
-----------------------------------------------------------------
time:   00:12:57
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579916/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579916/
sentry: https://sentry.ceph.com/organizations/ceph/?query=73dc41e47fb44d1fb66e7a9820645fb0

    Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)

[7579917]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/quiesce}
-----------------------------------------------------------------
time:   00:28:17
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579917/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579917/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d4555e10eb64971afe6359d486d4f6b

    Test failure: test_quiesce_path_release
    (tasks.cephfs.test_quiesce.TestQuiesce)

[7579918]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quiesce}
-----------------------------------------------------------------
time:   00:10:50
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579918/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579918/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e87f236f97f54af6958feb29eb0273fe

    Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)

[7579919]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/quiesce}
-----------------------------------------------------------------
time:   00:12:15
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579919/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579919/
sentry: https://sentry.ceph.com/organizations/ceph/?query=64fd3fdd1c564717ac8980685c6b6198

    Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)

[7579920]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quiesce}
-----------------------------------------------------------------
time:   00:22:47
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579920/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579920/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cd20846d9a504cba85e4baf971ac6bbf

    Test failure: test_quiesce_path_release
    (tasks.cephfs.test_quiesce.TestQuiesce)

[7579921]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/quiesce}
-----------------------------------------------------------------
time:   00:09:20
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579921/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579921/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d3b1f7be8b8e45efa5831497282444ce

    Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)

[7579922]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quiesce}
-----------------------------------------------------------------
time:   00:14:25
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579922/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579922/
sentry: https://sentry.ceph.com/organizations/ceph/?query=99fc0ebac75a4153a01b422fb95e301e

    Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)

[7579923]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/quiesce}
-----------------------------------------------------------------
time:   00:28:16
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579923/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579923/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d0885327f01048f0974241c6a1acd1f6

    Test failure: test_quiesce_path_release
    (tasks.cephfs.test_quiesce.TestQuiesce)

[7579924]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quiesce}
-----------------------------------------------------------------
time:   00:13:31
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579924/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579924/
sentry: https://sentry.ceph.com/organizations/ceph/?query=70246b7e6fa84560913770f96ff7afe8

    Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)

[7579925]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/quiesce}
-----------------------------------------------------------------
time:   00:14:12
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579925/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579925/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ae1d100c465c4dcd99695d3e1f1f02a0

    Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)

[7579926]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quiesce}
-----------------------------------------------------------------
time:   00:27:53
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579926/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579926/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2cc50ea0166f453787a1a32915a44135

    Test failure: test_quiesce_path_release
    (tasks.cephfs.test_quiesce.TestQuiesce)

[7579927]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/quiesce}
-----------------------------------------------------------------
time:   00:09:51
info:   https://pulpito.ceph.com/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579927/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2024-03-02_17:37:42-fs:functional-wip-batrick-testing-20240302.032523-distro-default-smithi/7579927/
sentry: https://sentry.ceph.com/organizations/ceph/?query=42e02c55100d496eb374d9d0b16f788a

    Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)