2023-01-20T23:24:13.636 INFO:root:teuthology version: 0.0.1.dev20+g5abecab
2023-01-20T23:24:13.637 INFO:teuthology.results:subset: '111/120000'
2023-01-20T23:24:13.637 INFO:teuthology.results:seed: '643'
2023-01-21T05:01:48.215 INFO:root:teuthology version: 0.0.1.dev20+g5abecab
2023-01-21T05:01:48.260 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-01-21T11:52:21.497 INFO:teuthology.results:Tests finished! gathering results...
2023-01-21T11:52:37.887 INFO:scrape:Found 186 jobs
2023-01-21T11:52:55.961 INFO:scrape:Found 13 distinct failure reasons
2023-01-21T11:52:55.962 INFO:scrape:Failure: "2023-01-21T03:21:39.354614+0000 mgr.x (mgr.14107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log
2023-01-21T11:52:55.963 INFO:scrape:12 jobs: ['7131266', '7131238', '7131172', '7131281', '7131187', '7131145', '7131156', '7131309', '7131293', '7131254', '7131216', '7131195']
2023-01-21T11:52:55.963 INFO:scrape:suites intersection: ['1-client', '2-upgrade', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus']
2023-01-21T11:52:55.964 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/quincy}}', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus']
2023-01-21T11:52:55.964 INFO:scrape:
2023-01-21T11:52:55.964 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-01-21T11:52:55.965 INFO:scrape:2 jobs: ['7131287', '7131255']
2023-01-21T11:52:55.965 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'objectstore-ec/bluestore-comp', 'osd}']
2023-01-21T11:52:55.966 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'session_timeout', 'session_timeout}', 'tasks/fsstress', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/lockdep}']
2023-01-21T11:52:55.966 INFO:scrape:
2023-01-21T11:52:55.966 INFO:scrape:Failure: Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays)
2023-01-21T11:52:55.967 INFO:scrape:1 jobs: ['7131311']
2023-01-21T11:52:55.967 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}']
2023-01-21T11:52:55.967 INFO:scrape:
2023-01-21T11:52:55.968 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi138 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2023-01-21T11:52:55.968 INFO:scrape:3 jobs: ['7131136', '7131246', '7131312']
2023-01-21T11:52:55.969 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-01-21T11:52:55.969 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-01-21T11:52:55.969 INFO:scrape:
2023-01-21T11:52:55.970 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2023-01-21T11:52:55.970 INFO:scrape:1 jobs: ['7131204']
2023-01-21T11:52:55.971 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}']
2023-01-21T11:52:55.971 INFO:scrape:
2023-01-21T11:52:55.971 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2023-01-21T11:52:55.972 INFO:scrape:1 jobs: ['7131316']
2023-01-21T11:52:55.972 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}']
2023-01-21T11:52:55.972 INFO:scrape:
2023-01-21T11:52:55.973 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-01-21T11:52:55.973 INFO:scrape:1 jobs: ['7131140']
2023-01-21T11:52:55.974 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-01-21T11:52:55.974 INFO:scrape:
2023-01-21T11:52:55.974 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2023-01-21T11:52:55.975 INFO:scrape:2 jobs: ['7131207', '7131292']
2023-01-21T11:52:55.975 INFO:scrape:suites intersection: ['2-logrotate}']
2023-01-21T11:52:55.975 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/mirror-ha/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{whitelist_health}', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'supported-random-distro$/{centos_8}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'workloads/cephfs-mirror-ha-workunit}']
2023-01-21T11:52:55.976 INFO:scrape:
2023-01-21T11:52:55.976 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi040.front.sepia.ceph.com: ['type=AVC msg=audit(1674263298.993:194): avc:  denied  { node_bind } for  pid=1415 comm="ping" saddr=172.21.15.40 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1']
2023-01-21T11:52:55.977 INFO:scrape:1 jobs: ['7131196']
2023-01-21T11:52:55.977 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-01-21T11:52:55.977 INFO:scrape:
2023-01-21T11:52:55.978 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-01-21T11:52:55.978 INFO:scrape:1 jobs: ['7131189']
2023-01-21T11:52:55.979 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-01-21T11:52:55.979 INFO:scrape:
2023-01-21T11:52:55.979 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=510284b66513490445619d1430aa869868c71a09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2023-01-21T11:52:55.980 INFO:scrape:1 jobs: ['7131252']
2023-01-21T11:52:55.980 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}']
2023-01-21T11:52:55.981 INFO:scrape:
2023-01-21T11:52:55.981 INFO:scrape:Dead: b'2023-01-21T11:32:05.703 DEBUG:teuthology.exit:Finished running handlers'
2023-01-21T11:52:55.981 INFO:scrape:1 jobs: ['7131141']
2023-01-21T11:52:55.982 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/no}']
2023-01-21T11:52:55.982 INFO:scrape:
2023-01-21T11:52:55.982 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration)
2023-01-21T11:52:55.983 INFO:scrape:1 jobs: ['7131317']
2023-01-21T11:52:55.983 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/test_journal_migration}']
2023-01-21T11:52:55.983 INFO:scrape:
2023-01-21T11:52:55.992 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/
failed:  27
dead:    1
running: 0
waiting: 0
queued:  0
passed:  158


Fail
=================================================================
[7131136]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:27:09
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131136/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131136/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f453d9c127bd43e8b6f71b5ea34d58fe

    Command failed (workunit test kernel_untar_build.sh) on smithi138 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=510284b66513490445619d1430aa869868c71a09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7131140]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:48:57
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131140/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131140/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1967df12439f40e983afeaf6d37dc014

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi039 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=510284b66513490445619d1430aa869868c71a09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7131145]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:20:15
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131145/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131145/

    "2023-01-20T23:50:42.982040+0000 mgr.x (mgr.24102) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131156]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:26:55
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131156/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131156/

    "2023-01-21T00:20:30.489685+0000 mgr.x (mgr.14098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131172]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:27:24
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131172/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131172/

    "2023-01-21T00:51:40.241959+0000 mgr.x (mgr.14103) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131187]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:20:48
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131187/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131187/

    "2023-01-21T01:13:45.681761+0000 mgr.y (mgr.14108) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131189]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:29:18
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131189/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131189/
sentry: https://sentry.ceph.com/organizations/ceph/?query=081b93900e17435ea591640403e334fc

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi102
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=510284b66513490445619d1430aa869868c71a09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7131195]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:25:44
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131195/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131195/

    "2023-01-21T01:25:14.529952+0000 mon.a (mon.0) 268 : cluster [WRN] Health
    check failed: 1 mgr modules have recently crashed
    (RECENT_MGR_MODULE_CRASH)" in cluster log

[7131196]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:51:03
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131196/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131196/

    SELinux denials found on ubuntu@smithi040.front.sepia.ceph.com: ['type=AVC
    msg=audit(1674263298.993:194): avc:  denied  { node_bind } for  pid=1415
    comm="ping" saddr=172.21.15.40 scontext=system_u:system_r:ping_t:s0
    tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1']

[7131204]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
-----------------------------------------------------------------
time:   00:13:10
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131204/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131204/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b95d60dd4f3f40289a574c0b563abded

    Test failure: test_client_cache_size
    (tasks.cephfs.test_client_limits.TestClientLimits)

[7131207]  fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit}
-----------------------------------------------------------------
time:   00:24:48
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131207/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131207/

    reached maximum tries (50) after waiting for 300 seconds

[7131216]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:27:37
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131216/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131216/

    "2023-01-21T01:55:29.763440+0000 mgr.x (mgr.14101) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131238]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:19:25
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131238/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131238/

    "2023-01-21T02:25:46.610014+0000 mgr.y (mgr.14109) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131246]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:33:00
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131246/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131246/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c09cb8d35fb64010b6c07d8bf404eaea

    Command failed (workunit test kernel_untar_build.sh) on smithi107 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=510284b66513490445619d1430aa869868c71a09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7131252]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:34:05
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131252/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131252/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4dd109f744a7484a9518ac0805885bfb

    Command failed (workunit test suites/pjd.sh) on smithi050 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=510284b66513490445619d1430aa869868c71a09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[7131254]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:26:07
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131254/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131254/

    "2023-01-21T02:51:14.781119+0000 mgr.x (mgr.14106) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131255]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   06:21:32
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131255/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131255/
sentry: https://sentry.ceph.com/organizations/ceph/?query=79152db980524984b266fe602dd92d0d

    Command failed (workunit test suites/fsstress.sh) on smithi170 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=510284b66513490445619d1430aa869868c71a09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7131266]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:27:32
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131266/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131266/

    "2023-01-21T03:21:39.354614+0000 mgr.x (mgr.14107) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131281]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:20:07
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131281/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131281/

    "2023-01-21T03:42:31.063198+0000 mgr.y (mgr.14111) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131287]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   06:22:01
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131287/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131287/
sentry: https://sentry.ceph.com/organizations/ceph/?query=374c2b105e7a42569bc048b18fd5254f

    Command failed (workunit test suites/fsstress.sh) on smithi123 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=510284b66513490445619d1430aa869868c71a09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7131292]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}}
-----------------------------------------------------------------
time:   01:22:17
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131292/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131292/

    error during scrub thrashing: reached maximum tries (30) after waiting for
    900 seconds

[7131293]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
-----------------------------------------------------------------
time:   00:25:14
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131293/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131293/

    "2023-01-21T04:10:26.415858+0000 mgr.x (mgr.14100) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131309]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:27:19
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131309/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131309/

    "2023-01-21T04:48:26.012021+0000 mgr.x (mgr.14098) 1 : cluster [ERR] Failed
    to load ceph-mgr modules: prometheus" in cluster log

[7131311]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
-----------------------------------------------------------------
time:   00:21:05
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131311/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131311/
sentry: https://sentry.ceph.com/organizations/ceph/?query=841c4d70664d438ca2b62e42ad096655

    Test failure: test_hardlink_reintegration
    (tasks.cephfs.test_strays.TestStrays)

[7131312]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:32:57
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131312/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131312/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d2814d4e9004f8ea7d7a1d5dfbd04e5

    Command failed (workunit test kernel_untar_build.sh) on smithi138 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=510284b66513490445619d1430aa869868c71a09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7131316]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:44:14
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131316/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131316/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eb86aea7f1c54a8390fecb463578a4a2

    Command failed (workunit test suites/dbench.sh) on smithi032 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=510284b66513490445619d1430aa869868c71a09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7131317]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:16:04
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131317/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131317/
sentry: https://sentry.ceph.com/organizations/ceph/?query=299f927263b949859811d05d025e987f

    Test failure: test_journal_migration
    (tasks.cephfs.test_journal_migration.TestJournalMigration)



Dead
=================================================================
[7131141]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131141/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131141/

    hit max job timeout



Pass
=================================================================
[7131137] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:31
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131137/

[7131138] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:42:44
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131138/

[7131139] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:34:04
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131139/

[7131142] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind}
time:   00:51:18
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131142/

[7131143] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
time:   00:47:42
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131143/

[7131144] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:17:45
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131144/

[7131146] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:12:07
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131146/

[7131147] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:11:57
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131147/

[7131148] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:25:25
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131148/

[7131149] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:37:01
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131149/

[7131150] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:43:57
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131150/

[7131151] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}}
time:   01:04:55
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131151/

[7131152] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:32:32
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131152/

[7131153] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:17:17
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131153/

[7131154] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:30:30
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131154/

[7131155] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:13:49
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131155/

[7131157] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:19:04
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131157/

[7131158] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}}
time:   00:59:32
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131158/

[7131159] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:30
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131159/

[7131160] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:11:10
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131160/

[7131161] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:10
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131161/

[7131162] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:22:34
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131162/

[7131163] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:15:44
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131163/

[7131164] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:33:56
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131164/

[7131165] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:22:57
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131165/

[7131166] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:14:16
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131166/

[7131167] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd}
time:   00:27:40
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131167/

[7131168] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}}
time:   00:24:59
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131168/

[7131169] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:13:32
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131169/

[7131170] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:51:12
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131170/

[7131171] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:15:34
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131171/

[7131173] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:12:48
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131173/

[7131174] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep}
time:   00:21:17
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131174/

[7131175] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:21:44
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131175/

[7131176] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:13:44
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131176/

[7131177] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:56:51
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131177/

[7131178] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:05:22
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131178/

[7131179] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:13:44
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131179/

[7131180] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:23
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131180/

[7131181] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:06:30
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131181/

[7131182] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}}
time:   00:42:12
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131182/

[7131183] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:07
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131183/

[7131184] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:15:21
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131184/

[7131185] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:23:45
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131185/

[7131186] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:13:18
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131186/

[7131188] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:30:47
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131188/

[7131190] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}}
time:   00:28:50
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131190/

[7131191] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:34:50
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131191/

[7131192] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:13:46
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131192/

[7131193] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:22:30
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131193/

[7131194] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   00:12:09
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131194/

[7131197] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:18:52
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131197/

[7131198] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}}
time:   00:35:54
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131198/

[7131199] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:27
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131199/

[7131200] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   00:12:58
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131200/

[7131201] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:48:22
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131201/

[7131202] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common}
time:   00:35:46
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131202/

[7131203] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:20:22
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131203/

[7131205] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:12:23
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131205/

[7131206] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror}
time:   01:58:46
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131206/

[7131208] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:41:56
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131208/

[7131209] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:51:46
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131209/

[7131210] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:35:31
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131210/

[7131211] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:14:38
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131211/

[7131212] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:42:10
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131212/

[7131213] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:57:49
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131213/

[7131214] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop}
time:   00:10:26
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131214/

[7131215] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:20:50
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131215/

[7131217] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:08:31
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131217/

[7131218] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:19:16
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131218/

[7131219] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   02:07:49
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131219/

[7131220] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:12:19
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131220/

[7131221] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind}
time:   00:51:08
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131221/

[7131222] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:16:56
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131222/

[7131223] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:34:38
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131223/

[7131224] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}}
time:   00:33:17
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131224/

[7131225] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:41:36
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131225/

[7131226] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:18:23
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131226/

[7131227] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:22:32
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131227/

[7131228] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:11:57
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131228/

[7131229] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   00:27:23
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131229/

[7131230] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:13:02
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131230/

[7131231] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}}
time:   00:57:11
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131231/

[7131232] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
time:   00:12:22
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131232/

[7131233] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:58:22
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131233/

[7131234] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:18
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131234/

[7131235] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:13:54
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131235/

[7131236] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:17:27
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131236/

[7131237] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:16:46
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131237/

[7131239] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}}
time:   00:25:04
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131239/

[7131240] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:20:19
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131240/

[7131241] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:20:06
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131241/

[7131242] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:12:13
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131242/

[7131243] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:54:35
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131243/

[7131244] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:52:56
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131244/

[7131245] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:32:26
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131245/

[7131247] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:51:16
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131247/

[7131248] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:21:38
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131248/

[7131249] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:26:19
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131249/

[7131250] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench}
time:   01:12:27
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131250/

[7131251] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:12:36
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131251/

[7131253] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:16:54
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131253/

[7131256] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:22:45
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131256/

[7131257] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:15:56
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131257/

[7131258] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:12:22
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131258/

[7131259] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:12:57
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131259/

[7131260] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:34:51
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131260/

[7131261] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:18:28
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131261/

[7131262] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:34:02
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131262/

[7131263] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}}
time:   00:23:33
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131263/

[7131264] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:07:36
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131264/

[7131265] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:29:03
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131265/

[7131267] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:18:52
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131267/

[7131268] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:15:13
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131268/

[7131269] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}}
time:   01:26:10
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131269/

[7131270] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:19:35
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131270/

[7131271] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:18:20
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131271/

[7131272] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:31:13
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131272/

[7131273] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:49:09
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131273/

[7131274] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   02:17:34
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131274/

[7131275] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:48:37
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131275/

[7131276] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:13:00
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131276/

[7131277] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:58
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131277/

[7131278] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:45:11
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131278/

[7131279] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:00
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131279/

[7131280] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:48:48
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131280/

[7131282] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:41:30
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131282/

[7131283] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   00:12:54
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131283/

[7131284] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}}
time:   00:39:29
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131284/

[7131285] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb}
time:   02:02:21
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131285/

[7131286] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:16:13
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131286/

[7131288] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:34:45
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131288/

[7131289] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
time:   00:50:53
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131289/

[7131290] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   00:22:04
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131290/

[7131291] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:18:45
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131291/

[7131294] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:15:20
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131294/

[7131295] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:13:06
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131295/

[7131296] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
time:   00:56:29
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131296/

[7131297] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:13:44
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131297/

[7131298] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:20:32
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131298/

[7131299] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:12:57
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131299/

[7131300] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:14:38
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131300/

[7131301] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:52:24
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131301/

[7131302] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:53:01
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131302/

[7131303] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:17
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131303/

[7131304] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:07:26
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131304/

[7131305] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   00:52:18
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131305/

[7131306] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:17:10
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131306/

[7131307] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}}
time:   00:38:58
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131307/

[7131308] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:22:14
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131308/

[7131310] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:20:11
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131310/

[7131313] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:16:52
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131313/

[7131314] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:12:13
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131314/

[7131315] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:18:02
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131315/

[7131318] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:12:43
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131318/

[7131319] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:16:56
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131319/

[7131320] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone}
time:   02:36:59
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131320/

[7131321] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
time:   00:12:19
info:   http://pulpito.ceph.com/yuriw-2023-01-20_17:21:59-fs-main-distro-default-smithi/7131321/