2024-03-14T08:46:39.292 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-14T08:46:39.292 INFO:teuthology.results:seed: '971'
2024-03-14T08:58:01.662 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-14T08:58:01.668 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-14T09:28:03.974 INFO:teuthology.results:Tests finished! gathering results...
2024-03-14T09:28:04.673 INFO:scrape:Found 12 jobs
2024-03-14T09:28:08.203 INFO:scrape:Found 1 distinct failure reasons
2024-03-14T09:28:08.204 INFO:scrape:Failure: Test failure: test_per_client_labeled_perf_counters_on_client_reconnect (tasks.cephfs.test_admin.TestLabeledPerfCounters)
2024-03-14T09:28:08.204 INFO:scrape:12 jobs: ['7599955', '7599949', '7599944', '7599947', '7599945', '7599948', '7599951', '7599946', '7599953', '7599950', '7599952', '7599954']
2024-03-14T09:28:08.205 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/admin}']
2024-03-14T09:28:08.205 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'tasks/admin}']
2024-03-14T09:28:08.205 INFO:scrape:
2024-03-14T09:28:08.206 INFO:teuthology.results:Sending results to jcollin@redhat.com: Test Run: jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/
failed:  12
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7599944]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/admin}
-----------------------------------------------------------------
time:   00:12:44
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599944/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599944/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b9a806542c43424292e5ac55d119c2ff

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599945]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/admin}
-----------------------------------------------------------------
time:   00:18:24
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599945/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599945/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d667a02034da4ee9a68239ea6efd7b60

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599946]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/admin}
-----------------------------------------------------------------
time:   00:16:09
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599946/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599946/
sentry: https://sentry.ceph.com/organizations/ceph/?query=05ae306ad055453ebbe6da3936ec6596

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599947]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin}
-----------------------------------------------------------------
time:   00:12:40
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599947/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599947/
sentry: https://sentry.ceph.com/organizations/ceph/?query=499c690785e1498fb83d4dd45eb52b54

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599948]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin}
-----------------------------------------------------------------
time:   00:13:19
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599948/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599948/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3f83c00ee72e4f3e83d1ec581fab2a6f

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599949]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin}
-----------------------------------------------------------------
time:   00:18:33
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599949/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599949/
sentry: https://sentry.ceph.com/organizations/ceph/?query=90e5f89a625f46fe966731bb32c67acd

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599950]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/admin}
-----------------------------------------------------------------
time:   00:12:06
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599950/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599950/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b80a06654086454bba291e8c03322a72

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599951]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/admin}
-----------------------------------------------------------------
time:   00:17:44
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599951/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599951/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f20bcc23a06541aa9e6482d23ef26d4c

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599952]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/admin}
-----------------------------------------------------------------
time:   00:16:39
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599952/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599952/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ec0d00e64f1a44dfa9d533402758ffa5

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599953]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin}
-----------------------------------------------------------------
time:   00:12:17
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599953/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599953/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d252246924ed4cdc84f6da6354492fb7

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599954]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin}
-----------------------------------------------------------------
time:   00:17:19
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599954/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599954/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c4f4f5f2f2f2465ca1557bf6106bca24

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)

[7599955]  fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin}
-----------------------------------------------------------------
time:   00:16:53
info:   https://pulpito.ceph.com/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599955/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2024-03-14_08:46:05-fs:functional-main-distro-default-smithi/7599955/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d3d20892173a415981e1c3175308089c

    Test failure: test_per_client_labeled_perf_counters_on_client_reconnect
    (tasks.cephfs.test_admin.TestLabeledPerfCounters)