2024-03-30T12:47:01.179 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-30T12:47:01.180 INFO:teuthology.results:seed: '364'
2024-03-30T12:52:56.807 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-30T12:52:56.810 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-30T13:16:58.794 INFO:teuthology.results:Tests finished! gathering results...
2024-03-30T13:16:59.359 INFO:scrape:Found 12 jobs
2024-03-30T13:17:01.662 INFO:scrape:Found 1 distinct failure reasons
2024-03-30T13:17:01.663 INFO:scrape:Failure: Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining)
2024-03-30T13:17:01.663 INFO:scrape:12 jobs: ['7629148', '7629153', '7629146', '7629152', '7629143', '7629150', '7629147', '7629149', '7629154', '7629151', '7629145', '7629144']
2024-03-30T13:17:01.663 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no_client_pidfile', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/uninlining}']
2024-03-30T13:17:01.663 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'tasks/uninlining}']
2024-03-30T13:17:01.664 INFO:scrape:
2024-03-30T13:17:01.664 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/
failed:  12
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7629143]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/uninlining}
-----------------------------------------------------------------
time:   00:11:03
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629143/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629143/
sentry: https://sentry.ceph.com/organizations/ceph/?query=41d4e3628d4b40c4932ded0d16d4a4c2

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629144]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/uninlining}
-----------------------------------------------------------------
time:   00:15:01
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629144/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629144/
sentry: https://sentry.ceph.com/organizations/ceph/?query=07875d0cb1b147bf875dd90c403953d7

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629145]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/uninlining}
-----------------------------------------------------------------
time:   00:16:45
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629145/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629145/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5acdc2b8bfb2401d909c5088f33eb0ec

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629146]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining}
-----------------------------------------------------------------
time:   00:10:30
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629146/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629146/
sentry: https://sentry.ceph.com/organizations/ceph/?query=51bd58fd45194d6eb60bb00e9cfdcd30

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629147]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining}
-----------------------------------------------------------------
time:   00:11:21
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629147/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629147/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d3894de0abcb4370b7125db64b377d5b

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629148]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining}
-----------------------------------------------------------------
time:   00:15:10
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629148/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629148/
sentry: https://sentry.ceph.com/organizations/ceph/?query=60ea57b241c9427fb83096af644adb38

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629149]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/uninlining}
-----------------------------------------------------------------
time:   00:13:02
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629149/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629149/
sentry: https://sentry.ceph.com/organizations/ceph/?query=79f3f4b5ec5841e8b898c6ef751a59d2

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629150]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/uninlining}
-----------------------------------------------------------------
time:   00:15:36
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629150/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629150/
sentry: https://sentry.ceph.com/organizations/ceph/?query=33bf6c56bd6b4b00811a359c831529f9

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629151]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/uninlining}
-----------------------------------------------------------------
time:   00:15:52
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629151/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629151/
sentry: https://sentry.ceph.com/organizations/ceph/?query=278ad8a289bd4d3fb2077933870cec12

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629152]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining}
-----------------------------------------------------------------
time:   00:13:20
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629152/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629152/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f7042268376e49dab853fa8408f5cc76

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629153]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining}
-----------------------------------------------------------------
time:   00:11:28
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629153/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629153/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d6ede4d19a974d158d50c767a92d4cf0

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)

[7629154]  fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/uninlining}
-----------------------------------------------------------------
time:   00:14:39
info:   https://pulpito.ceph.com/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629154/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-03-29_13:38:54-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629154/
sentry: https://sentry.ceph.com/organizations/ceph/?query=36431adacf1c43ef911b3b0550fdf5bb

    Test failure: test_data_uninlining
    (tasks.cephfs.test_uninlining.TestDataUninlining)