2023-03-25T13:30:51.219 INFO:root:teuthology version: 0.0.1.dev1+ge372916
2023-03-25T13:30:51.220 INFO:teuthology.results:subset: '816/1024'
2023-03-25T13:30:51.220 INFO:teuthology.results:seed: '9480'
2023-03-25T21:51:12.043 INFO:root:teuthology version: 0.0.1.dev1+ge372916
2023-03-25T21:51:12.070 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-03-26T09:39:18.295 INFO:teuthology.results:Tests finished! gathering results...
2023-03-26T09:39:40.844 INFO:scrape:Found 216 jobs
2023-03-26T09:39:50.476 INFO:scrape:Found 11 distinct failure reasons
2023-03-26T09:39:50.477 INFO:scrape:Dead: b'2023-03-26T01:36:53.934 DEBUG:teuthology.exit:Finished running handlers'
2023-03-26T09:39:50.478 INFO:scrape:ceph version 18.0.0-3066-gf03d69a8 (f03d69a8741c605ae6711afe2c9991ac46095f7d) reef (dev)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f086fe12cf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x5592df62fd8b]
 5: (KernelDevice::_aio_thread()+0x1798) [0x5592e023f908]
 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5592e0241fa1]
 7: /lib64/libpthread.so.0(+0x81ca) [0x7f086fe081ca]
 8: clone()
2023-03-26T09:39:50.478 INFO:scrape:1 jobs: ['7213413']
2023-03-26T09:39:50.478 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-03-26T09:39:50.479 INFO:scrape:
2023-03-26T09:39:50.479 INFO:scrape:Timeout 120 running ceph --cluster ceph tell mds.1:0 scrub status
2023-03-26T09:39:50.479 INFO:scrape:1 jobs: ['7213540']
2023-03-26T09:39:50.480 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}']
2023-03-26T09:39:50.481 INFO:scrape:
2023-03-26T09:39:50.481 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
2023-03-26T09:39:50.482 INFO:scrape:1 jobs: ['7213546']
2023-03-26T09:39:50.482 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific']
2023-03-26T09:39:50.483 INFO:scrape:
2023-03-26T09:39:50.484 INFO:scrape:Dead: b'2023-03-26T09:27:49.920 DEBUG:teuthology.exit:Finished running handlers'
2023-03-26T09:39:50.484 INFO:scrape:2 jobs: ['7213615', '7213479']
2023-03-26T09:39:50.485 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-03-26T09:39:50.486 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'frag', 'fs/functional/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/data-scan}', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/no}']
2023-03-26T09:39:50.486 INFO:scrape:
2023-03-26T09:39:50.487 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-03-26T09:39:50.488 INFO:scrape:3 jobs: ['7213436', '7213506', '7213519']
2023-03-26T09:39:50.488 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}']
2023-03-26T09:39:50.489 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'session_timeout}', 'tasks/fsstress', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/valgrind}']
2023-03-26T09:39:50.490 INFO:scrape:
2023-03-26T09:39:50.490 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-03-26T09:39:50.491 INFO:scrape:2 jobs: ['7213517', '7213572']
2023-03-26T09:39:50.491 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay', 'wsync/no}']
2023-03-26T09:39:50.492 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/blogbench}}', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}']
2023-03-26T09:39:50.492 INFO:scrape:
2023-03-26T09:39:50.493 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2023-03-26T09:39:50.494 INFO:scrape:1 jobs: ['7213559']
2023-03-26T09:39:50.494 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{centos_8}', 'workloads/cephfs-mirror-ha-workunit}']
2023-03-26T09:39:50.494 INFO:scrape:
2023-03-26T09:39:50.495 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2023-03-26T09:39:50.495 INFO:scrape:1 jobs: ['7213595']
2023-03-26T09:39:50.495 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-03-26T09:39:50.496 INFO:scrape:
2023-03-26T09:39:50.496 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f03d69a8741c605ae6711afe2c9991ac46095f7d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2023-03-26T09:39:50.496 INFO:scrape:1 jobs: ['7213464']
2023-03-26T09:39:50.497 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}']
2023-03-26T09:39:50.497 INFO:scrape:
2023-03-26T09:39:50.497 INFO:scrape:Failure: Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage)
2023-03-26T09:39:50.498 INFO:scrape:1 jobs: ['7213474']
2023-03-26T09:39:50.498 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/damage}']
2023-03-26T09:39:50.498 INFO:scrape:
2023-03-26T09:39:50.499 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi099 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f03d69a8741c605ae6711afe2c9991ac46095f7d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2023-03-26T09:39:50.499 INFO:scrape:2 jobs: ['7213575', '7213497']
2023-03-26T09:39:50.499 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-03-26T09:39:50.499 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'frag', 'fs/mixed-clients/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2023-03-26T09:39:50.500 INFO:scrape:
2023-03-26T09:39:50.512 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/
failed:  13
dead:    3
running: 0
waiting: 0
queued:  0
passed:  200


Fail
=================================================================
[7213436]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   06:28:58
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213436/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213436/
sentry: https://sentry.ceph.com/organizations/ceph/?query=45dd45afb761429c802ded61334653e8

    Command failed (workunit test suites/fsstress.sh) on smithi176 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f03d69a8741c605ae6711afe2c9991ac46095f7d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7213464]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:33:16
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213464/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213464/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d009d85ed81a4246b0567575a19897e6

    Command failed (workunit test suites/pjd.sh) on smithi007 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f03d69a8741c605ae6711afe2c9991ac46095f7d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7213474]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
-----------------------------------------------------------------
time:   00:28:53
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213474/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213474/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6fc96a38c8c04cec9cb78a213505fa41

    Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage)

[7213497]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:42:56
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213497/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213497/
sentry: https://sentry.ceph.com/organizations/ceph/?query=99659156a63d4fd196285e37af04017c

    Command failed (workunit test kernel_untar_build.sh) on smithi045 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f03d69a8741c605ae6711afe2c9991ac46095f7d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'

[7213506]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   06:17:12
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213506/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213506/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e0d00bc0a6bf4d4897108b26373230cb

    Command failed (workunit test suites/fsstress.sh) on smithi098 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f03d69a8741c605ae6711afe2c9991ac46095f7d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7213517]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:41:56
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213517/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213517/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7213519]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   06:29:44
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213519/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213519/
sentry: https://sentry.ceph.com/organizations/ceph/?query=90556eadcfea4e7fb405c1ce05a38cb1

    Command failed (workunit test suites/fsstress.sh) on smithi026 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f03d69a8741c605ae6711afe2c9991ac46095f7d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7213540]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:42:46
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213540/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213540/

    error during scrub thrashing: Command failed on smithi110 with status 124:
    'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
    timeout 120 ceph --cluster ceph tell mds.1:0 scrub status'

[7213546]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   03:42:20
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213546/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213546/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1767caa84b8b456ab891319ee783ef27

    Command failed (workunit test suites/fsstress.sh) on smithi122 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f03d69a8741c605ae6711afe2c9991ac46095f7d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[7213559]  fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit}
-----------------------------------------------------------------
time:   00:46:29
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213559/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213559/

    reached maximum tries (50) after waiting for 300 seconds

[7213572]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:38:51
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213572/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213572/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7213575]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:26:27
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213575/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213575/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c8c8d2858f624ad6bf1d3c85229e1df3

    Command failed (workunit test kernel_untar_build.sh) on smithi099 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f03d69a8741c605ae6711afe2c9991ac46095f7d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7213595]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   11:27:07
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213595/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213595/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9394c2a1cfc540e589a2e4d652277c95

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)



Dead
=================================================================
[7213413]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213413/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213413/

    hit max job timeout

[7213479]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213479/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213479/

    hit max job timeout

[7213615]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213615/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213615/

    hit max job timeout



Pass
=================================================================
[7213414] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/blogbench}}
time:   00:29:04
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213414/

[7213415] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:20:08
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213415/

[7213416] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:00:15
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213416/

[7213417] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:44:33
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213417/

[7213418] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   02:00:29
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213418/

[7213419] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}}
time:   00:34:28
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213419/

[7213420] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:03:46
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213420/

[7213421] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:13:58
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213421/

[7213422] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:29:01
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213422/

[7213423] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:23:30
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213423/

[7213424] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/dbench}}
time:   00:57:25
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213424/

[7213425] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:14:29
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213425/

[7213426] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:21:24
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213426/

[7213427] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:23:37
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213427/

[7213428] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}}
time:   00:38:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213428/

[7213429] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/fsstress}}
time:   00:26:12
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213429/

[7213430] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   01:03:51
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213430/

[7213431] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:12:39
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213431/

[7213432] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb}
time:   02:04:26
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213432/

[7213433] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:34
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213433/

[7213434] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:18:07
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213434/

[7213435] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:20:11
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213435/

[7213437] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:32:09
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213437/

[7213438] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}}
time:   00:56:42
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213438/

[7213439] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:22:29
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213439/

[7213440] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:20:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213440/

[7213441] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/iozone}}
time:   00:23:45
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213441/

[7213442] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:15:34
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213442/

[7213443] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:37:39
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213443/

[7213444] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:25:59
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213444/

[7213445] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}}
time:   01:53:53
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213445/

[7213446] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:19:58
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213446/

[7213447] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   00:20:05
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213447/

[7213448] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}}
time:   00:36:19
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213448/

[7213449] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:52
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213449/

[7213450] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:46
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213450/

[7213451] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:25:45
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213451/

[7213452] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:21:36
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213452/

[7213453] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:08:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213453/

[7213454] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:48:29
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213454/

[7213455] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:21:46
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213455/

[7213456] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   02:02:35
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213456/

[7213457] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:17:24
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213457/

[7213458] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:15:46
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213458/

[7213459] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:12:44
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213459/

[7213460] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:40:29
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213460/

[7213461] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:22:25
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213461/

[7213462] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:24:51
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213462/

[7213463] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:19:02
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213463/

[7213465] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/new_ops/client-sanity}}
time:   00:18:48
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213465/

[7213466] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:20:46
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213466/

[7213467] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:26:08
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213467/

[7213468] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/blogbench}}
time:   00:29:01
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213468/

[7213469] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:35:01
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213469/

[7213470] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:17:22
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213470/

[7213471] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:37:39
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213471/

[7213472] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone}
time:   02:25:43
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213472/

[7213473] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}}
time:   00:36:59
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213473/

[7213475] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:20:17
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213475/

[7213476] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:29:25
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213476/

[7213477] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:27:01
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213477/

[7213478] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:23:23
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213478/

[7213480] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep}
time:   00:57:45
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213480/

[7213481] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/dbench}}
time:   01:02:30
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213481/

[7213482] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   02:02:09
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213482/

[7213483] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:45:45
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213483/

[7213484] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}}
time:   00:43:40
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213484/

[7213485] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/fsstress}}
time:   00:25:23
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213485/

[7213486] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:33:52
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213486/

[7213487] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:31:13
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213487/

[7213488] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:17:22
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213488/

[7213489] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:56:44
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213489/

[7213490] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:38:50
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213490/

[7213491] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:20:55
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213491/

[7213492] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   01:02:26
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213492/

[7213493] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:21:18
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213493/

[7213494] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:30:17
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213494/

[7213495] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   01:21:47
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213495/

[7213496] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:32:20
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213496/

[7213498] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:19:41
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213498/

[7213499] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/iozone}}
time:   00:24:05
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213499/

[7213500] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:31:21
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213500/

[7213501] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}}
time:   01:52:23
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213501/

[7213502] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:14:41
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213502/

[7213503] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:23:10
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213503/

[7213504] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:19:43
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213504/

[7213505] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:22:50
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213505/

[7213507] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}}
time:   01:17:05
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213507/

[7213508] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:36:44
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213508/

[7213509] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:32:52
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213509/

[7213510] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:17:21
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213510/

[7213511] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/new_ops/client-sanity}}
time:   00:20:21
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213511/

[7213512] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:15:17
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213512/

[7213513] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd}
time:   00:20:49
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213513/

[7213514] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:22
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213514/

[7213515] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:17:03
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213515/

[7213516] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:34:23
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213516/

[7213518] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/blogbench}}
time:   00:30:24
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213518/

[7213520] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:21:48
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213520/

[7213521] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:31:05
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213521/

[7213522] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:59:23
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213522/

[7213523] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:37:50
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213523/

[7213524] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:28:06
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213524/

[7213525] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:21:26
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213525/

[7213526] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:29:48
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213526/

[7213527] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:51:25
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213527/

[7213528] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:56:18
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213528/

[7213529] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:15:55
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213529/

[7213530] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:18:17
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213530/

[7213531] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/dbench}}
time:   00:58:35
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213531/

[7213532] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}}
time:   01:14:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213532/

[7213533] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:26:12
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213533/

[7213534] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:24:08
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213534/

[7213535] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:17:11
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213535/

[7213536] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:18:54
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213536/

[7213537] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   00:20:27
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213537/

[7213538] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/fsstress}}
time:   00:25:47
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213538/

[7213539] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:59:50
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213539/

[7213541] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:32:17
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213541/

[7213542] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:12:40
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213542/

[7213543] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/iozone}}
time:   00:24:14
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213543/

[7213544] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   00:51:05
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213544/

[7213545] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   00:25:16
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213545/

[7213547] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}}
time:   00:32:55
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213547/

[7213548] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:27:39
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213548/

[7213549] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:16:15
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213549/

[7213550] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}}
time:   01:56:26
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213550/

[7213551] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
time:   00:50:31
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213551/

[7213552] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:25:12
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213552/

[7213553] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   00:19:41
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213553/

[7213554] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:53:31
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213554/

[7213555] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common}
time:   02:23:34
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213555/

[7213556] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:27:46
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213556/

[7213557] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:19:24
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213557/

[7213558] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror}
time:   02:26:24
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213558/

[7213560] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:06:33
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213560/

[7213561] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:47:45
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213561/

[7213562] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:20:30
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213562/

[7213563] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:54:26
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213563/

[7213564] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:17:24
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213564/

[7213565] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop}
time:   00:18:58
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213565/

[7213566] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:34:15
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213566/

[7213567] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   02:29:34
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213567/

[7213568] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:10:33
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213568/

[7213569] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:37:32
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213569/

[7213570] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:16:08
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213570/

[7213571] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep}
time:   00:58:25
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213571/

[7213573] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:19:21
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213573/

[7213574] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:17:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213574/

[7213576] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   01:02:28
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213576/

[7213577] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:44:17
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213577/

[7213578] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/new_ops/client-sanity}}
time:   00:19:11
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213578/

[7213579] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:09:29
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213579/

[7213580] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:31:48
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213580/

[7213581] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}}
time:   00:32:37
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213581/

[7213582] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:17:04
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213582/

[7213583] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:22:31
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213583/

[7213584] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/blogbench}}
time:   00:29:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213584/

[7213585] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:28:33
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213585/

[7213586] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:30:17
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213586/

[7213587] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:21:17
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213587/

[7213588] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:23:32
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213588/

[7213589] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:23:54
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213589/

[7213590] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/dbench}}
time:   01:03:24
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213590/

[7213591] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}}
time:   00:37:02
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213591/

[7213592] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:14:09
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213592/

[7213593] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:24:45
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213593/

[7213594] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/fsstress}}
time:   00:25:47
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213594/

[7213596] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:18:18
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213596/

[7213597] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:53:09
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213597/

[7213598] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:52:45
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213598/

[7213599] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   00:56:05
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213599/

[7213600] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:17:47
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213600/

[7213601] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:39:40
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213601/

[7213602] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:19:08
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213602/

[7213603] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}}
time:   00:40:57
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213603/

[7213604] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:34:32
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213604/

[7213605] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:18:42
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213605/

[7213606] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench}
time:   01:17:53
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213606/

[7213607] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:21:08
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213607/

[7213608] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/iozone}}
time:   00:23:22
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213608/

[7213609] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:51:37
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213609/

[7213610] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:17:36
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213610/

[7213611] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:24:47
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213611/

[7213612] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
time:   01:26:44
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213612/

[7213613] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:05:00
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213613/

[7213614] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}}
time:   01:52:39
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213614/

[7213616] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:18:05
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213616/

[7213617] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:22:31
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213617/

[7213618] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:19:02
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213618/

[7213619] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:22:40
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213619/

[7213620] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:15:42
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213620/

[7213621] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:32:48
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213621/

[7213622] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:42:41
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213622/

[7213623] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   01:08:20
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213623/

[7213624] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}}
time:   00:25:53
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213624/

[7213625] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/new_ops/client-sanity}}
time:   00:19:47
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213625/

[7213626] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:14:05
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213626/

[7213627] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   01:08:07
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213627/

[7213628] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/blogbench}}
time:   00:30:09
info:   http://pulpito.ceph.com/vshankar-2023-03-18_05:01:47-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7213628/