2023-11-15T10:13:04.251 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-15T10:13:04.252 INFO:teuthology.results:subset: '965/1024'
2023-11-15T10:13:04.252 INFO:teuthology.results:seed: '1253'
2023-11-15T14:35:34.109 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-15T14:35:34.144 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-15T22:55:16.883 INFO:teuthology.results:Tests finished! gathering results...
2023-11-15T22:55:34.172 INFO:scrape:Found 204 jobs
2023-11-15T22:55:35.396 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458973/teuthology.log
2023-11-15T22:55:45.183 INFO:scrape:Found 13 distinct failure reasons
2023-11-15T22:55:45.184 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi061 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2c87685078fe6089375b9becf37dec26d5d30cc2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2023-11-15T22:55:45.184 INFO:scrape:1 jobs: ['7459098']
2023-11-15T22:55:45.185 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2023-11-15T22:55:45.185 INFO:scrape:
2023-11-15T22:55:45.186 INFO:scrape:Failure: Command failed on smithi079 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib'
2023-11-15T22:55:45.186 INFO:scrape:1 jobs: ['7458971']
2023-11-15T22:55:45.187 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}']
2023-11-15T22:55:45.187 INFO:scrape:
2023-11-15T22:55:45.187 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2c87685078fe6089375b9becf37dec26d5d30cc2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-11-15T22:55:45.188 INFO:scrape:1 jobs: ['7459060']
2023-11-15T22:55:45.188 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/snaps}']
2023-11-15T22:55:45.189 INFO:scrape:
2023-11-15T22:55:45.189 INFO:scrape:Dead: None
2023-11-15T22:55:45.190 INFO:scrape:2 jobs: ['7458973', '7459005']
2023-11-15T22:55:45.190 INFO:scrape:
2023-11-15T22:55:45.190 INFO:scrape:Failure: Command failed on smithi153 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2c87685078fe6089375b9becf37dec26d5d30cc2 pull'
2023-11-15T22:55:45.191 INFO:scrape:1 jobs: ['7458924']
2023-11-15T22:55:45.191 INFO:scrape:suites: ['fs/nfs/{cluster/{1-node}', 'overrides/ignorelist_health', 'supported-random-distros$/{centos_latest}', 'tasks/nfs}']
2023-11-15T22:55:45.192 INFO:scrape:
2023-11-15T22:55:45.192 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2023-11-15T22:55:45.192 INFO:scrape:1 jobs: ['7458929']
2023-11-15T22:55:45.193 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2023-11-15T22:55:45.193 INFO:scrape:
2023-11-15T22:55:45.193 INFO:scrape:Failure: "2023-11-15T14:32:01.789871+0000 mds.b (mds.1) 1 : cluster [WRN] client.4984 isn't responding to mclientcaps(revoke), ino 0x200000028ed pending pAsLsXsFsc issued pAsLsXsFscb, sent 300.004000 seconds ago" in cluster log
2023-11-15T22:55:45.194 INFO:scrape:1 jobs: ['7459055']
2023-11-15T22:55:45.194 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2023-11-15T22:55:45.195 INFO:scrape:
2023-11-15T22:55:45.195 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
2023-11-15T22:55:45.196 INFO:scrape:1 jobs: ['7459077']
2023-11-15T22:55:45.196 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}']
2023-11-15T22:55:45.196 INFO:scrape:
2023-11-15T22:55:45.197 INFO:scrape:Failure: Test failure: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
2023-11-15T22:55:45.197 INFO:scrape:1 jobs: ['7458940']
2023-11-15T22:55:45.198 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}']
2023-11-15T22:55:45.198 INFO:scrape:
2023-11-15T22:55:45.198 INFO:scrape:Dead: b'2023-11-15T22:44:53.789 DEBUG:teuthology.exit:Finished running handlers'
2023-11-15T22:55:45.199 INFO:scrape:1 jobs: ['7458917']
2023-11-15T22:55:45.199 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}']
2023-11-15T22:55:45.200 INFO:scrape:
2023-11-15T22:55:45.200 INFO:scrape:Failure: Command failed on smithi196 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7b12d75c-83c5-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json'
2023-11-15T22:55:45.200 INFO:scrape:1 jobs: ['7459089']
2023-11-15T22:55:45.201 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific']
2023-11-15T22:55:45.201 INFO:scrape:
2023-11-15T22:55:45.201 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)
2023-11-15T22:55:45.202 INFO:scrape:1 jobs: ['7459046']
2023-11-15T22:55:45.202 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}']
2023-11-15T22:55:45.203 INFO:scrape:
2023-11-15T22:55:45.203 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2023-11-15T22:55:45.203 INFO:scrape:1 jobs: ['7459007']
2023-11-15T22:55:45.204 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']
2023-11-15T22:55:45.204 INFO:scrape:
2023-11-15T22:55:45.215 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi
=================================================================
info:    https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/
failed:  11
dead:    3
running: 0
waiting: 0
queued:  0
passed:  190


Fail
=================================================================
[7458924]  fs/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs}
-----------------------------------------------------------------
time:   00:13:08
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458924/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458924/
sentry: https://sentry.ceph.com/organizations/ceph/?query=23558c19acd946028f7ea5526a4870a2

    Command failed on smithi153 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:2c87685078fe6089375b9becf37dec26d5d30cc2 pull'

[7458929]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   01:30:06
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458929/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458929/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22d7454cce704e0aafc42b5a28b08abd

    Test failure: test_cephfs_mirror_restart_sync_on_blocklist
    (tasks.cephfs.test_mirroring.TestMirroring)

[7458940]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
-----------------------------------------------------------------
time:   00:19:42
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458940/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458940/
sentry: https://sentry.ceph.com/organizations/ceph/?query=99d03a0a7e3a43a58da3697ed25c812b

    Test failure: test_client_blocklisted_oldest_tid
    (tasks.cephfs.test_client_limits.TestClientLimits)

[7458971]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:13:39
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458971/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458971/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7a39fa6de5a548a2ae853478a79f178a

    Command failed on smithi079 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1
    librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-
    mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib'

[7459007]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
-----------------------------------------------------------------
time:   00:17:19
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459007/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459007/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cfb714e5b3b84120ad7165e20be027e0

    Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

[7459046]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
-----------------------------------------------------------------
time:   00:36:36
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459046/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459046/
sentry: https://sentry.ceph.com/organizations/ceph/?query=362120d7900043e1aa31f33066a00220

    Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

[7459055]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   05:32:14
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459055/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459055/

    "2023-11-15T14:32:01.789871+0000 mds.b (mds.1) 1 : cluster [WRN]
    client.4984 isn't responding to mclientcaps(revoke), ino 0x200000028ed
    pending pAsLsXsFsc issued pAsLsXsFscb, sent 300.004000 seconds ago" in
    cluster log

[7459060]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:24:58
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459060/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459060/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d5a7c0594fdf4972908763db69bb81d2

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi121 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2c87685078fe6089375b9becf37dec26d5d30cc2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7459077]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}}
-----------------------------------------------------------------
time:   00:55:00
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459077/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459077/

    error during scrub thrashing: reached maximum tries (31) after waiting for
    900 seconds

[7459089]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:17:26
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459089/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459089/
sentry: https://sentry.ceph.com/organizations/ceph/?query=25060ddfdf224b8398e9154c75f902ec

    Command failed on smithi196 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    7b12d75c-83c5-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json'

[7459098]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:51:01
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459098/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459098/
sentry: https://sentry.ceph.com/organizations/ceph/?query=35593bfffa9e432daa5ff669e1c7a806

    Command failed (workunit test suites/dbench.sh) on smithi061 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2c87685078fe6089375b9becf37dec26d5d30cc2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'



Dead
=================================================================
[7458917]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458917/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458917/

    hit max job timeout

[7458973]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458973/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458973/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds

[7459005]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459005/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459005/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds



Pass
=================================================================
[7458903] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:14:08
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458903/

[7458904] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:24:07
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458904/

[7458905] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep}
time:   00:58:22
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458905/

[7458906] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:15:35
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458906/

[7458907] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:27:52
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458907/

[7458908] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}}
time:   00:53:05
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458908/

[7458909] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:21:07
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458909/

[7458910] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:27:43
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458910/

[7458911] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:17:28
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458911/

[7458912] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:17:20
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458912/

[7458913] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:37:00
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458913/

[7458914] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:22:04
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458914/

[7458915] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   00:17:14
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458915/

[7458916] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:52:45
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458916/

[7458918] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:26:07
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458918/

[7458919] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:16:59
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458919/

[7458920] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror}
time:   02:15:09
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458920/

[7458921] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_20.04} workloads/cephfs-mirror-ha-workunit}
time:   00:37:50
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458921/

[7458922] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:01:31
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458922/

[7458923] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:52:48
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458923/

[7458925] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:20:53
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458925/

[7458926] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:50:33
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458926/

[7458927] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:52:03
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458927/

[7458928] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/fstop}
time:   00:17:37
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458928/

[7458930] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   00:14:47
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458930/

[7458931] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:34:10
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458931/

[7458932] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}}
time:   00:45:39
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458932/

[7458933] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:24:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458933/

[7458934] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:15:44
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458934/

[7458935] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:17:18
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458935/

[7458936] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   02:14:55
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458936/

[7458937] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:51:53
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458937/

[7458938] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:51:16
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458938/

[7458939] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}}
time:   00:18:02
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458939/

[7458941] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:25:12
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458941/

[7458942] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}}
time:   01:13:54
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458942/

[7458943] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:19:20
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458943/

[7458944] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:14:52
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458944/

[7458945] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:45:56
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458945/

[7458946] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:39:29
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458946/

[7458947] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   01:00:03
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458947/

[7458948] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}}
time:   01:01:11
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458948/

[7458949] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:27:36
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458949/

[7458950] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:45
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458950/

[7458951] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:35
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458951/

[7458952] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:35:50
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458952/

[7458953] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:33:37
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458953/

[7458954] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
time:   01:27:27
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458954/

[7458955] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   00:45:25
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458955/

[7458956] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:30:23
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458956/

[7458957] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}}
time:   00:24:28
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458957/

[7458958] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:17:00
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458958/

[7458959] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:26:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458959/

[7458960] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:53:10
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458960/

[7458961] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}}
time:   00:28:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458961/

[7458962] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:52:31
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458962/

[7458963] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:23:34
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458963/

[7458964] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:37:32
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458964/

[7458965] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench}
time:   01:11:39
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458965/

[7458966] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:42:20
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458966/

[7458967] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:19:27
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458967/

[7458968] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:18:23
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458968/

[7458969] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:27:16
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458969/

[7458970] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:22:38
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458970/

[7458972] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:22:27
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458972/

[7458974] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:59:14
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458974/

[7458975] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:22:33
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458975/

[7458976] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:22:03
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458976/

[7458977] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:16:24
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458977/

[7458978] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:19:18
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458978/

[7458979] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:23:00
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458979/

[7458980] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:16:48
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458980/

[7458981] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:37:30
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458981/

[7458982] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}}
time:   00:46:40
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458982/

[7458983] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458983/

[7458984] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:25:26
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458984/

[7458985] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:34:46
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458985/

[7458986] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}}
time:   01:37:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458986/

[7458987] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:18:56
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458987/

[7458988] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:23:06
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458988/

[7458989] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:17:26
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458989/

[7458990] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:50:52
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458990/

[7458991] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:41:22
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458991/

[7458992] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep}
time:   00:55:08
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458992/

[7458993] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:53:09
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458993/

[7458994] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}}
time:   00:41:21
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458994/

[7458995] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:27:02
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458995/

[7458996] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:32:11
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458996/

[7458997] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:53:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458997/

[7458998] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:39:05
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458998/

[7458999] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:21:59
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7458999/

[7459000] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:23:15
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459000/

[7459001] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:30
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459001/

[7459002] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:05
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459002/

[7459003] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}}
time:   00:36:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459003/

[7459004] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:17:29
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459004/

[7459006] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   01:15:54
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459006/

[7459008] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:52:27
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459008/

[7459009] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:24:04
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459009/

[7459010] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:27:22
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459010/

[7459011] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:16:10
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459011/

[7459012] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   00:16:26
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459012/

[7459013] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}}
time:   00:41:24
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459013/

[7459014] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:01
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459014/

[7459015] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:28:57
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459015/

[7459016] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:19:07
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459016/

[7459017] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   00:58:33
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459017/

[7459018] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}}
time:   01:00:12
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459018/

[7459019] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:16:04
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459019/

[7459020] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   00:35:45
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459020/

[7459021] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}}
time:   00:36:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459021/

[7459022] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:22:27
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459022/

[7459023] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:24:14
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459023/

[7459024] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:31:17
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459024/

[7459025] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:01:18
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459025/

[7459026] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:42:24
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459026/

[7459027] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:58
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459027/

[7459028] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:21:23
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459028/

[7459029] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:17:34
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459029/

[7459030] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:32:32
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459030/

[7459031] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:34:47
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459031/

[7459032] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
time:   01:35:47
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459032/

[7459033] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
time:   01:54:15
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459033/

[7459034] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:17:26
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459034/

[7459035] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:22:59
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459035/

[7459036] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}}
time:   00:40:22
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459036/

[7459037] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:17:11
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459037/

[7459038] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:39:07
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459038/

[7459039] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:25:17
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459039/

[7459040] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:27:43
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459040/

[7459041] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:20:09
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459041/

[7459042] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone}
time:   02:26:47
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459042/

[7459043] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:16:55
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459043/

[7459044] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}}
time:   00:26:55
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459044/

[7459045] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}}
time:   00:21:42
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459045/

[7459047] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:47:19
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459047/

[7459048] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:26:16
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459048/

[7459049] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   01:43:53
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459049/

[7459050] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:09
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459050/

[7459051] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:23:31
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459051/

[7459052] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:26:56
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459052/

[7459053] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:23:22
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459053/

[7459054] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:57
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459054/

[7459056] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:15:10
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459056/

[7459057] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}}
time:   01:21:21
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459057/

[7459058] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:15:21
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459058/

[7459059] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:50:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459059/

[7459061] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:16:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459061/

[7459062] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:23:55
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459062/

[7459063] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:27:24
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459063/

[7459064] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:18:20
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459064/

[7459065] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:49:29
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459065/

[7459066] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}}
time:   02:20:18
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459066/

[7459067] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:15:41
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459067/

[7459068] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep}
time:   00:56:59
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459068/

[7459069] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:25:14
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459069/

[7459070] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:21:51
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459070/

[7459071] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:37:15
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459071/

[7459072] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:22:56
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459072/

[7459073] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:21:27
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459073/

[7459074] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:16:56
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459074/

[7459075] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:55
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459075/

[7459076] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}}
time:   00:28:11
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459076/

[7459078] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:23:05
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459078/

[7459079] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:19:32
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459079/

[7459080] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:34:03
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459080/

[7459081] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:17:11
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459081/

[7459082] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:25:15
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459082/

[7459083] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd}
time:   00:19:59
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459083/

[7459084] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:37:09
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459084/

[7459085] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:52:28
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459085/

[7459086] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}}
time:   00:43:11
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459086/

[7459087] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:20:23
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459087/

[7459088] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:16:25
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459088/

[7459090] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:33:50
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459090/

[7459091] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:30:23
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459091/

[7459092] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:00:50
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459092/

[7459093] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   00:19:55
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459093/

[7459094] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:22:19
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459094/

[7459095] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:26:58
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459095/

[7459096] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:14:10
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459096/

[7459097] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:38:32
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459097/

[7459099] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:26:02
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459099/

[7459100] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:22:38
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459100/

[7459101] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:23:03
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459101/

[7459102] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:28:37
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459102/

[7459103] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:17:38
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459103/

[7459104] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:37:01
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459104/

[7459105] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:16:26
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459105/

[7459106] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   01:06:31
info:   https://pulpito.ceph.com/vshankar-2023-11-15_09:32:33-fs-wip-vshankar-testing2-2023-11-15-1128-reef-testing-default-smithi/7459106/