2024-02-28T15:18:15.415 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-28T15:18:15.415 INFO:teuthology.results:seed: '4485' 2024-02-28T15:19:23.111 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-28T15:19:23.115 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-28T23:18:01.927 INFO:teuthology.results:Tests finished! gathering results... 2024-02-28T23:18:07.785 INFO:scrape:Found 60 jobs 2024-02-28T23:18:10.215 INFO:scrape:Found 4 distinct failure reasons 2024-02-28T23:18:10.216 INFO:scrape:Failure: Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan) 2024-02-28T23:18:10.216 INFO:scrape:1 jobs: ['7575561'] 2024-02-28T23:18:10.216 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/data-scan}'] 2024-02-28T23:18:10.216 INFO:scrape: 2024-02-28T23:18:10.216 INFO:scrape:Failure: Command failed on smithi193 with status 1: 'sudo yum install -y kernel' 2024-02-28T23:18:10.216 INFO:scrape:1 jobs: ['7575520'] 2024-02-28T23:18:10.216 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/damage}'] 2024-02-28T23:18:10.216 INFO:scrape: 2024-02-28T23:18:10.216 INFO:scrape:Failure: Failed to reconnect to smithi193 2024-02-28T23:18:10.216 INFO:scrape:1 jobs: ['7575519'] 2024-02-28T23:18:10.217 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/alternate-pool}'] 2024-02-28T23:18:10.217 INFO:scrape: 2024-02-28T23:18:10.217 INFO:scrape:Crash: SELinux denials found on ubuntu@smithi053.front.sepia.ceph.com: ['type=AVC msg=audit(1709134973.724:199): avc: denied { checkpoint_restore } for pid=1282 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-02-28T23:18:10.217 INFO:scrape:ceph version 19.0.0-1737-gbb8b899a (bb8b899a7a08efb407a126f2c49434118c007b01) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fc014254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fc0142a154c] 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7fc014b815d4] 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x559d6fd0f4e1] 7: (CDentry::check_corruption(bool)+0x48d) [0x559d6ff1ee4d] 8: ceph-mds(+0x37c03c) [0x559d6ff3103c] 9: (CDir::_omap_commit(int)+0x93a) [0x559d6ff3859a] 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x559d6ff2f895] 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xa61) [0x559d6fd543e1] 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x559d6ffb8c82] 13: (MDLog::trim_all()+0x708) [0x559d6ffbbc78] 14: ceph-mds(+0x18dec1) [0x559d6fd42ec1] 15: ceph-mds(+0x133dbd) [0x559d6fce8dbd] 16: (MDSContext::complete(int)+0x5c) [0x559d6ffb1d7c] 17: ceph-mds(+0x134062) [0x559d6fce9062] 18: (MDSContext::complete(int)+0x5c) [0x559d6ffb1d7c] 19: (MDSIOContextBase::complete(int)+0x31c) [0x559d6ffb21cc] 20: (Finisher::finisher_thread_entry()+0x175) [0x7fc014c2c955] 21: /lib64/libc.so.6(+0x9f802) [0x7fc01429f802] 22: /lib64/libc.so.6(+0x3f450) [0x7fc01423f450] 2024-02-28T23:18:10.217 INFO:scrape:1 jobs: ['7575525'] 2024-02-28T23:18:10.217 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/damage}'] 2024-02-28T23:18:10.217 INFO:scrape: 2024-02-28T23:18:10.220 INFO:teuthology.results:Sending results to jcollin@redhat.com: Test Run: jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 56 Fail ================================================================= [7575519] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/alternate-pool} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575519/ log: http://qa-proxy.ceph.com/teuthology/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575519/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75ee6b097d384e358e3891ef71eb8139 Failed to reconnect to smithi193 [7575520] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575520/ log: http://qa-proxy.ceph.com/teuthology/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575520/ sentry: https://sentry.ceph.com/organizations/ceph/?query=935b91b85561447aa2da23047a47dc4c Command failed on smithi193 with status 1: 'sudo yum install -y kernel' [7575525] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} ----------------------------------------------------------------- time: 00:25:36 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575525/ log: http://qa-proxy.ceph.com/teuthology/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575525/ SELinux denials found on ubuntu@smithi053.front.sepia.ceph.com: ['type=AVC msg=audit(1709134973.724:199): avc: denied { checkpoint_restore } for pid=1282 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7575561] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/data-scan} ----------------------------------------------------------------- time: 00:32:13 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575561/ log: http://qa-proxy.ceph.com/teuthology/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575561/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45d1408db5e046aa88ee315894220cd3 Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan) Pass ================================================================= [7575514] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/alternate-pool} time: 00:13:33 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575514/ [7575515] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:20:53 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575515/ [7575516] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 00:45:27 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575516/ [7575517] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} time: 00:20:05 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575517/ [7575518] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:24:15 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575518/ [7575521] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 00:39:56 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575521/ [7575522] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} time: 00:21:12 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575522/ [7575523] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:23:25 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575523/ [7575524] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/alternate-pool} time: 00:14:40 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575524/ [7575526] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 03:12:03 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575526/ [7575527] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} time: 00:23:12 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575527/ [7575528] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:17:10 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575528/ [7575529] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:14:08 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575529/ [7575530] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:20:40 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575530/ [7575531] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/data-scan} time: 00:44:01 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575531/ [7575532] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} time: 00:19:59 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575532/ [7575533] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} time: 00:24:09 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575533/ [7575534] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:10:51 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575534/ [7575535] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:22:03 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575535/ [7575536] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/data-scan} time: 00:40:32 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575536/ [7575537] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} time: 00:24:47 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575537/ [7575538] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} time: 00:21:54 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575538/ [7575539] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:13:49 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575539/ [7575540] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:24:27 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575540/ [7575541] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/data-scan} time: 07:46:09 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575541/ [7575542] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} time: 00:24:29 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575542/ [7575543] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} time: 00:17:15 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575543/ [7575544] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/alternate-pool} time: 00:14:49 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575544/ [7575545] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:20:43 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575545/ [7575546] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 00:41:51 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575546/ [7575547] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} time: 00:19:44 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575547/ [7575548] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:18:27 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575548/ [7575549] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/alternate-pool} time: 00:10:50 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575549/ [7575550] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:22:31 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575550/ [7575551] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 00:39:14 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575551/ [7575552] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} time: 00:20:54 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575552/ [7575553] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:21:43 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575553/ [7575554] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/alternate-pool} time: 00:10:44 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575554/ [7575555] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:26:29 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575555/ [7575556] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 02:28:32 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575556/ [7575557] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} time: 00:21:23 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575557/ [7575558] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:17:22 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575558/ [7575559] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:16:57 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575559/ [7575560] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:22:50 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575560/ [7575562] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} time: 00:20:29 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575562/ [7575563] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} time: 00:23:56 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575563/ [7575564] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:11:15 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575564/ [7575565] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:22:46 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575565/ [7575566] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/data-scan} time: 00:39:15 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575566/ [7575567] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} time: 00:23:20 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575567/ [7575568] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} time: 00:20:40 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575568/ [7575569] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:14:24 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575569/ [7575570] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:25:12 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575570/ [7575571] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/data-scan} time: 02:13:02 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575571/ [7575572] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} time: 00:22:30 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575572/ [7575573] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} time: 00:17:27 info: https://pulpito.ceph.com/jcollin-2024-02-28_15:18:03-fs:functional-wip-jcollin-cephfs-journal-tool-testing-distro-default-smithi/7575573/