2023-08-30T07:50:12.666 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-30T07:50:12.667 INFO:teuthology.results:subset: '630/1024'
2023-08-30T07:50:12.667 INFO:teuthology.results:seed: '8279'
2023-08-30T09:05:44.979 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-30T09:05:45.005 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-08-30T20:20:32.202 INFO:teuthology.results:Tests finished! gathering results...
2023-08-30T20:20:39.337 INFO:scrape:Found 65 jobs
2023-08-30T20:20:51.017 INFO:scrape:Found 8 distinct failure reasons
2023-08-30T20:20:51.018 INFO:scrape:Failure: 'NoneType' object has no attribute '_fields'
2023-08-30T20:20:51.019 INFO:scrape:10 jobs: ['7383837', '7383796', '7383839', '7383804', '7383853', '7383825', '7383810', '7383824', '7383820', '7383801']
2023-08-30T20:20:51.019 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}']
2023-08-30T20:20:51.020 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/admin}', 'tasks/cephfs_misc_tests}', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/ior-shared-file}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/mdtest}', 'tasks/test_journal_migration}', 'tasks/xfstests-dev}', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'traceless/50pc}']
2023-08-30T20:20:51.020 INFO:scrape:
2023-08-30T20:20:51.021 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=89881124a82fb6f56a558633d5da2bcf1545c766 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2023-08-30T20:20:51.021 INFO:scrape:1 jobs: ['7383800']
2023-08-30T20:20:51.021 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}']
2023-08-30T20:20:51.022 INFO:scrape:
2023-08-30T20:20:51.022 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-08-30T20:20:51.023 INFO:scrape:3 jobs: ['7383794', '7383830', '7383798']
2023-08-30T20:20:51.023 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay', 'wsync/yes}']
2023-08-30T20:20:51.024 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/fsync-tester}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2023-08-30T20:20:51.025 INFO:scrape:
2023-08-30T20:20:51.025 INFO:scrape:Failure: 
2023-08-30T20:20:51.025 INFO:scrape:8 jobs: ['7383822', '7383805', '7383806', '7383852', '7383803', '7383814', '7383799', '7383850']
2023-08-30T20:20:51.026 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{centos_latest}', 'mds', 'mon', 'osd}']
2023-08-30T20:20:51.026 INFO:scrape:suites union: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/suites/iozone}}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/pool-perm}', 'tasks/volumes/{overrides', 'tasks/{0-client', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/misc}}', 'thrashosds-health}', 'traceless/50pc}']
2023-08-30T20:20:51.027 INFO:scrape:
2023-08-30T20:20:51.027 INFO:scrape:Failure: Test failure: test_volume_rm (tasks.cephfs.test_volumes.TestVolumes)
2023-08-30T20:20:51.027 INFO:scrape:1 jobs: ['7383791']
2023-08-30T20:20:51.028 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}']
2023-08-30T20:20:51.028 INFO:scrape:
2023-08-30T20:20:51.028 INFO:scrape:Dead: b'2023-08-30T20:17:09.674 DEBUG:teuthology.exit:Finished running handlers'
2023-08-30T20:20:51.029 INFO:scrape:2 jobs: ['7383821', '7383797']
2023-08-30T20:20:51.029 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2023-08-30T20:20:51.029 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_20.04}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/multifs-auth}']
2023-08-30T20:20:51.030 INFO:scrape:
2023-08-30T20:20:51.030 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
2023-08-30T20:20:51.031 INFO:scrape:1 jobs: ['7383807']
2023-08-30T20:20:51.031 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}']
2023-08-30T20:20:51.031 INFO:scrape:
2023-08-30T20:20:51.032 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2023-08-30T20:20:51.032 INFO:scrape:1 jobs: ['7383790']
2023-08-30T20:20:51.032 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2023-08-30T20:20:51.033 INFO:scrape:
2023-08-30T20:20:51.036 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/
failed:  25
dead:    2
running: 0
waiting: 0
queued:  0
passed:  38


Fail
=================================================================
[7383790]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   01:33:19
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383790/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383790/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ad58f4d3dceb4af4b278b57e132e7e61

    Test failure: test_cephfs_mirror_restart_sync_on_blocklist
    (tasks.cephfs.test_mirroring.TestMirroring)

[7383791]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:35:38
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383791/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383791/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e19cac6bf3d4c29a66ccd3541f32657

    Test failure: test_volume_rm (tasks.cephfs.test_volumes.TestVolumes)

[7383794]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   01:15:14
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383794/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383794/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7383796]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383796/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383796/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a5d146ff2d474e68921e98ba77110edd

    'NoneType' object has no attribute '_fields'

[7383798]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:42:34
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383798/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383798/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7383799]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383799/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383799/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4b959053173a4125bef184798ad4d57a

[7383800]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:53:57
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383800/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383800/
sentry: https://sentry.ceph.com/organizations/ceph/?query=88d22b49bb33421b9dc1ff30d7cd6190

    Command failed (workunit test suites/ffsb.sh) on smithi038 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=89881124a82fb6f56a558633d5da2bcf1545c766
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

[7383801]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383801/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383801/
sentry: https://sentry.ceph.com/organizations/ceph/?query=999722426fb842b7a9da043ccc61921a

    'NoneType' object has no attribute '_fields'

[7383803]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383803/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383803/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ac2dd17f139340b89737d0d9837d9d44

[7383804]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383804/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383804/
sentry: https://sentry.ceph.com/organizations/ceph/?query=235e58d2670643e6abdb2b42722160dc

    'NoneType' object has no attribute '_fields'

[7383805]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383805/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383805/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c25b597befde4c208e5e8887c95ecd3c

[7383806]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383806/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383806/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cd68ac4d4ff94df2922f297babf77cb5

[7383807]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
-----------------------------------------------------------------
time:   00:37:08
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383807/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383807/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9784f5283ec140d5903e170d8cbf619e

    Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

[7383810]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383810/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383810/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bc2c957d098b44f1b9d66f81c9ad1678

    'NoneType' object has no attribute '_fields'

[7383814]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383814/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383814/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3c4675e9a89747f6a29e78abc4e5302f

[7383820]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383820/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383820/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13c5c2243fc741cc9d25c5d0c2b18f9e

    'NoneType' object has no attribute '_fields'

[7383822]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383822/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383822/
sentry: https://sentry.ceph.com/organizations/ceph/?query=24fe26d98e034b9b9a2ccfc7fbf78f87

[7383824]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383824/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383824/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5faf7f8849e941d283435d9b0a5eefd4

    'NoneType' object has no attribute '_fields'

[7383825]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383825/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383825/
sentry: https://sentry.ceph.com/organizations/ceph/?query=286e09c7f8a048609333ce878d842c16

    'NoneType' object has no attribute '_fields'

[7383830]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}}
-----------------------------------------------------------------
time:   00:40:27
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383830/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383830/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7383837]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383837/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383837/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a8b7b09f4fb14256a8a24be289dbb7c3

    'NoneType' object has no attribute '_fields'

[7383839]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383839/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383839/
sentry: https://sentry.ceph.com/organizations/ceph/?query=29c2f747f21a4a46b796142c8dd9c180

    'NoneType' object has no attribute '_fields'

[7383850]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383850/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383850/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0c97580896c24d38b95fc86713772ec0

[7383852]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383852/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383852/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2ef9a59fad8f448c894327df118b2e58

[7383853]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383853/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383853/
sentry: https://sentry.ceph.com/organizations/ceph/?query=929eca346bb4496f9ddb7e486ed2004d

    'NoneType' object has no attribute '_fields'



Dead
=================================================================
[7383797]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383797/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383797/

    hit max job timeout

[7383821]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383821/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383821/

    hit max job timeout



Pass
=================================================================
[7383792] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:03
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383792/

[7383793] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:16:41
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383793/

[7383795] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:26:07
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383795/

[7383802] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:25:53
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383802/

[7383808] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:21:11
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383808/

[7383809] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}}
time:   00:41:06
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383809/

[7383811] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:25:32
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383811/

[7383812] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   00:53:19
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383812/

[7383813] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}}
time:   00:53:54
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383813/

[7383815] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}}
time:   00:43:16
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383815/

[7383816] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:20:13
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383816/

[7383817] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:22:00
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383817/

[7383818] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:24:27
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383818/

[7383819] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:30:41
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383819/

[7383823] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:52:46
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383823/

[7383826] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:36:19
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383826/

[7383827] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:16:03
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383827/

[7383828] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:30:52
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383828/

[7383829] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:35:32
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383829/

[7383831] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:24:03
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383831/

[7383832] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
time:   00:57:29
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383832/

[7383833] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:18:15
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383833/

[7383834] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:15:55
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383834/

[7383835] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:22:53
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383835/

[7383836] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}}
time:   00:14:52
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383836/

[7383838] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:46:32
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383838/

[7383840] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}}
time:   00:36:13
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383840/

[7383841] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:16:56
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383841/

[7383842] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}}
time:   00:22:04
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383842/

[7383843] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:17:24
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383843/

[7383844] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:17:17
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383844/

[7383845] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:25:51
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383845/

[7383846] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:18:47
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383846/

[7383847] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:56
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383847/

[7383848] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:23:02
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383848/

[7383849] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}}
time:   00:40:47
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383849/

[7383851] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:18:11
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383851/

[7383854] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:25:56
info:   http://pulpito.ceph.com/vshankar-2023-08-30_05:46:16-fs-wip-vshankar-testing-20230825.182636-testing-default-smithi/7383854/