2023-05-09T07:30:44.467 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-09T07:30:44.487 INFO:teuthology.results:subset: '699/1024'
2023-05-09T07:30:44.487 INFO:teuthology.results:seed: '5770'
2023-05-09T17:15:01.892 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-09T17:15:01.929 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-09T20:52:22.126 INFO:teuthology.results:Tests finished! gathering results...
2023-05-09T20:52:39.646 INFO:scrape:Found 205 jobs
2023-05-09T20:52:56.321 WARNING:scrape:Misunderstood line: 
2023-05-09T20:53:14.010 INFO:scrape:Found 15 distinct failure reasons
2023-05-09T20:53:14.011 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-05-09T20:53:14.012 INFO:scrape:1 jobs: ['7268017']
2023-05-09T20:53:14.012 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}']
2023-05-09T20:53:14.012 INFO:scrape:
2023-05-09T20:53:14.013 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-05-09T20:53:14.013 INFO:scrape:4 jobs: ['7267969', '7267908', '7268021', '7268062']
2023-05-09T20:53:14.013 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay']
2023-05-09T20:53:14.014 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2023-05-09T20:53:14.014 INFO:scrape:
2023-05-09T20:53:14.014 INFO:scrape:Failure: Command failed on smithi184 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-05-09T20:53:14.015 INFO:scrape:4 jobs: ['7267936', '7267870', '7267971', '7268043']
2023-05-09T20:53:14.015 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health']
2023-05-09T20:53:14.015 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}']
2023-05-09T20:53:14.016 INFO:scrape:
2023-05-09T20:53:14.016 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench'
2023-05-09T20:53:14.016 INFO:scrape:3 jobs: ['7268049', '7267925', '7267884']
2023-05-09T20:53:14.017 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}']
2023-05-09T20:53:14.017 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-dbench}', 'tasks/fscrypt-ffsb}', 'tasks/fscrypt-iozone}']
2023-05-09T20:53:14.017 INFO:scrape:
2023-05-09T20:53:14.018 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh'
2023-05-09T20:53:14.018 INFO:scrape:1 jobs: ['7267895']
2023-05-09T20:53:14.018 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}']
2023-05-09T20:53:14.018 INFO:scrape:
2023-05-09T20:53:14.019 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-05-09T20:53:14.019 INFO:scrape:2 jobs: ['7267887', '7267906']
2023-05-09T20:53:14.019 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'mds', 'mon', 'mount/fuse', 'osd}']
2023-05-09T20:53:14.020 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-05-09T20:53:14.020 INFO:scrape:
2023-05-09T20:53:14.020 INFO:scrape:Failure: Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan)
2023-05-09T20:53:14.020 INFO:scrape:1 jobs: ['7267951']
2023-05-09T20:53:14.021 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/data-scan}']
2023-05-09T20:53:14.021 INFO:scrape:
2023-05-09T20:53:14.021 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
2023-05-09T20:53:14.022 INFO:scrape:2 jobs: ['7267982', '7267907']
2023-05-09T20:53:14.022 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific']
2023-05-09T20:53:14.022 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific']
2023-05-09T20:53:14.023 INFO:scrape:
2023-05-09T20:53:14.023 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable)
2023-05-09T20:53:14.023 INFO:scrape:1 jobs: ['7268023']
2023-05-09T20:53:14.023 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2023-05-09T20:53:14.024 INFO:scrape:
2023-05-09T20:53:14.024 INFO:scrape:Dead: b'2023-05-09T20:16:17.269 DEBUG:teuthology.exit:Finished running handlers'
2023-05-09T20:53:14.024 INFO:scrape:1 jobs: ['7267891']
2023-05-09T20:53:14.025 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-05-09T20:53:14.025 INFO:scrape:
2023-05-09T20:53:14.025 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
2023-05-09T20:53:14.025 INFO:scrape:1 jobs: ['7268003']
2023-05-09T20:53:14.026 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}']
2023-05-09T20:53:14.026 INFO:scrape:
2023-05-09T20:53:14.026 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi130 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'
2023-05-09T20:53:14.027 INFO:scrape:1 jobs: ['7267954']
2023-05-09T20:53:14.027 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}']
2023-05-09T20:53:14.027 INFO:scrape:
2023-05-09T20:53:14.027 INFO:scrape:Failure: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh'
2023-05-09T20:53:14.028 INFO:scrape:1 jobs: ['7268040']
2023-05-09T20:53:14.028 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-05-09T20:53:14.028 INFO:scrape:
2023-05-09T20:53:14.029 INFO:scrape:Failure: "1683631303.6575446 mon.a (mon.0) 2554 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log
2023-05-09T20:53:14.029 INFO:scrape:1 jobs: ['7268009']
2023-05-09T20:53:14.029 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2023-05-09T20:53:14.029 INFO:scrape:
2023-05-09T20:53:14.030 INFO:scrape:Failure: Command failed on smithi193 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2ef5137fbb6e28908d2161218e7b594628c1b6d3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6703c3c2-ee45-11ed-9b02-001a4aab830c -- ceph mon dump -f json'
2023-05-09T20:53:14.030 INFO:scrape:1 jobs: ['7267901']
2023-05-09T20:53:14.030 INFO:scrape:suites: ['1-rename', 'distro/single-container-host', 'fs/cephadm/renamevolume/{0-start', 'overrides/ignorelist_health}']
2023-05-09T20:53:14.031 INFO:scrape:
2023-05-09T20:53:14.045 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/
failed:  24
dead:    1
running: 0
waiting: 0
queued:  0
passed:  180


Fail
=================================================================
[7267870]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:16:40
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267870/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267870/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dc8dde227d6845a9ba109353be0c9498

    Command failed on smithi130 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7267884]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb}
-----------------------------------------------------------------
time:   00:11:16
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267884/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267884/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0645ea5deaba4e59bdeae3c3d2c13356

    Command failed (workunit test fs/fscrypt.sh) on smithi186 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none ffsb'

[7267887]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   01:21:32
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267887/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267887/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aea363d37a18448a95e44ce62d2b6cf3

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi031 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7267895]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:19:27
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267895/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267895/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0ade88efe28249e8b95e327e703ff8c1

    Command failed (workunit test fs/quota/quota.sh) on smithi093 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd --
    /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3
    CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh'

[7267901]  fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
-----------------------------------------------------------------
time:   00:12:53
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267901/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267901/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e6981914f7d84525acc14334872bba43

    Command failed on smithi193 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:2ef5137fbb6e28908d2161218e7b594628c1b6d3 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    6703c3c2-ee45-11ed-9b02-001a4aab830c -- ceph mon dump -f json'

[7267906]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:47:31
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267906/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267906/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0490cb97a91746cfa2d4b321342a3dea

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi125 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7267907]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   03:37:48
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267907/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267907/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e2a395f2d368472abf1090cd5a11ce70

    Command failed (workunit test suites/fsstress.sh) on smithi202 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[7267908]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   02:45:42
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267908/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267908/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7267925]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone}
-----------------------------------------------------------------
time:   00:33:49
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267925/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267925/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e63403a963674765b2df329d385c4d80

    Command failed (workunit test fs/fscrypt.sh) on smithi116 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none
    iozone'

[7267936]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:17:30
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267936/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267936/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e4229572c16497498d7907f5c3b90ad

    Command failed on smithi184 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7267951]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
-----------------------------------------------------------------
time:   06:01:39
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267951/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267951/
sentry: https://sentry.ceph.com/organizations/ceph/?query=47dcf7adeff341368abf12dd2271a5bf

    Test failure: test_rebuild_inotable
    (tasks.cephfs.test_data_scan.TestDataScan)

[7267954]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   00:21:17
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267954/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267954/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca7ec8be76bf401e80fa14dba9faa629

    Command failed (workunit test fs/test_python.sh) on smithi130 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'

[7267969]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:40:48
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267969/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267969/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7267971]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:17:47
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267971/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267971/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8698ccc49f9a44e4970c8e4ea7e3e666

    Command failed on smithi114 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7267982]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   03:37:56
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267982/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267982/
sentry: https://sentry.ceph.com/organizations/ceph/?query=48a4a9208ea141359e0348f4dd4480ff

    Command failed (workunit test suites/fsstress.sh) on smithi177 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[7268003]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common}
-----------------------------------------------------------------
time:   00:17:55
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268003/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268003/
sentry: https://sentry.ceph.com/organizations/ceph/?query=43082dff19a840f49ca38fff1889ab86

    Test failure: test_fscrypt_dummy_encryption_with_quick_group
    (tasks.cephfs.test_fscrypt.TestFscrypt)

[7268009]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:36:22
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268009/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268009/

    "1683631303.6575446 mon.a (mon.0) 2554 : cluster [WRN] Health check failed:
    Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)"
    in cluster log

[7268017]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   00:14:27
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268017/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268017/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4b127ee79954b62a054ddad01ce67e2

    Test failure: test_disable_enable_human_readable_quota_values
    (tasks.cephfs.test_quota.TestQuota)

[7268021]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:46:38
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268021/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268021/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7268023]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   02:00:01
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268023/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268023/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ade85d81719d431c9b252cd042ecd0e7

    saw valgrind issues

[7268040]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   03:51:11
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268040/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268040/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0450a54d7265440abd96ac7dfccb0862

    Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi070 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh'

[7268043]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:18:31
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268043/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268043/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d11f70f60f3d469cb21f0a12a0b16cc7

    Command failed on smithi177 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7268049]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench}
-----------------------------------------------------------------
time:   00:15:50
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268049/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268049/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aaa274eab75645af994416d08f9d7439

    Command failed (workunit test fs/fscrypt.sh) on smithi038 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ef5137fbb6e28908d2161218e7b594628c1b6d3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none
    dbench'

[7268062]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:53:33
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268062/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268062/

    error during scrub thrashing: rank damage found: {'backtrace'}



Dead
=================================================================
[7267891]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267891/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267891/

    hit max job timeout



Pass
=================================================================
[7267859] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:14:26
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267859/

[7267860] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:33:02
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267860/

[7267861] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind}
time:   00:53:31
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267861/

[7267862] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
time:   01:02:20
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267862/

[7267863] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:41:10
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267863/

[7267864] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}}
time:   00:39:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267864/

[7267865] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:21:42
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267865/

[7267866] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:10:59
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267866/

[7267867] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}}
time:   01:33:49
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267867/

[7267868] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:20:42
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267868/

[7267869] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:32:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267869/

[7267871] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:25:33
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267871/

[7267872] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:47:02
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267872/

[7267873] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:33:56
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267873/

[7267874] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
time:   00:28:42
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267874/

[7267875] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:25:05
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267875/

[7267876] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:34:47
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267876/

[7267877] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:20:19
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267877/

[7267878] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:21:49
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267878/

[7267879] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:55
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267879/

[7267880] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   02:02:54
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267880/

[7267881] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:36:52
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267881/

[7267882] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:36:52
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267882/

[7267883] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:17:02
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267883/

[7267885] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:29:42
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267885/

[7267886] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}}
time:   00:38:08
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267886/

[7267888] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:21:12
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267888/

[7267889] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:33:03
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267889/

[7267890] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:55:42
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267890/

[7267892] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:33:03
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267892/

[7267893] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}}
time:   00:45:20
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267893/

[7267894] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:22:50
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267894/

[7267896] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}}
time:   00:47:36
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267896/

[7267897] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep}
time:   00:21:38
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267897/

[7267898] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:19:32
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267898/

[7267899] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:27:23
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267899/

[7267900] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:54
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267900/

[7267902] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:30:07
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267902/

[7267903] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:00:09
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267903/

[7267904] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:53:57
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267904/

[7267905] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:25
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267905/

[7267909] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:13:24
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267909/

[7267910] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:23:54
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267910/

[7267911] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:46:04
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267911/

[7267912] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:18:36
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267912/

[7267913] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:33:37
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267913/

[7267914] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:19:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267914/

[7267915] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:14:01
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267915/

[7267916] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:23:33
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267916/

[7267917] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}}
time:   00:57:43
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267917/

[7267918] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:22:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267918/

[7267919] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}}
time:   00:08:23
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267919/

[7267920] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:55:57
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267920/

[7267921] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:51:23
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267921/

[7267922] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:15:46
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267922/

[7267923] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}}
time:   00:35:29
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267923/

[7267924] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:20:36
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267924/

[7267926] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   00:18:57
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267926/

[7267927] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
time:   00:52:32
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267927/

[7267928] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:40:40
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267928/

[7267929] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:14:31
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267929/

[7267930] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:23:24
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267930/

[7267931] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:23:21
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267931/

[7267932] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:21:57
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267932/

[7267933] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:48:53
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267933/

[7267934] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:30:12
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267934/

[7267935] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:29:11
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267935/

[7267937] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:30:23
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267937/

[7267938] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:47:59
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267938/

[7267939] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind}
time:   01:02:07
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267939/

[7267940] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:54:03
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267940/

[7267941] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}}
time:   01:11:16
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267941/

[7267942] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:24:07
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267942/

[7267943] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:51:52
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267943/

[7267944] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:45:09
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267944/

[7267945] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:36:44
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267945/

[7267946] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:41:44
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267946/

[7267947] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:21:17
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267947/

[7267948] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:37:20
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267948/

[7267949] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}}
time:   01:04:09
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267949/

[7267950] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:58:36
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267950/

[7267952] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}}
time:   00:38:53
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267952/

[7267953] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:36
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267953/

[7267955] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:34
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267955/

[7267956] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:37:33
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267956/

[7267957] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   01:01:36
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267957/

[7267958] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:50:09
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267958/

[7267959] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   02:17:48
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267959/

[7267960] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:49:11
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267960/

[7267961] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}}
time:   00:32:52
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267961/

[7267962] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:25:13
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267962/

[7267963] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:33:20
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267963/

[7267964] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:41:42
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267964/

[7267965] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd}
time:   00:23:34
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267965/

[7267966] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:01:51
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267966/

[7267967] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:27:20
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267967/

[7267968] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:31:17
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267968/

[7267970] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:20:15
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267970/

[7267972] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:58:06
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267972/

[7267973] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:16:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267973/

[7267974] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:24:37
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267974/

[7267975] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:15:12
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267975/

[7267976] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:38:21
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267976/

[7267977] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep}
time:   00:22:08
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267977/

[7267978] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:20:23
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267978/

[7267979] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:23:27
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267979/

[7267980] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:20:54
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267980/

[7267981] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:31:07
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267981/

[7267983] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}}
time:   00:33:13
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267983/

[7267984] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:16:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267984/

[7267985] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:25:56
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267985/

[7267986] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
time:   00:32:44
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267986/

[7267987] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:23:08
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267987/

[7267988] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:13:27
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267988/

[7267989] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}}
time:   00:50:34
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267989/

[7267990] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:28:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267990/

[7267991] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:24:45
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267991/

[7267992] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:50:13
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267992/

[7267993] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:24:04
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267993/

[7267994] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:19:52
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267994/

[7267995] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:33:31
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267995/

[7267996] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:15:18
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267996/

[7267997] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   02:08:38
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267997/

[7267998] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}}
time:   00:43:20
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267998/

[7267999] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:15:37
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7267999/

[7268000] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:23
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268000/

[7268001] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   00:15:24
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268001/

[7268002] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:45:16
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268002/

[7268004] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:25:20
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268004/

[7268005] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:21:24
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268005/

[7268006] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror}
time:   02:08:54
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268006/

[7268007] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit}
time:   00:27:04
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268007/

[7268008] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:00:33
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268008/

[7268010] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:19:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268010/

[7268011] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:48:59
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268011/

[7268012] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:07:52
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268012/

[7268013] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/fstop}
time:   00:21:11
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268013/

[7268014] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:16:51
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268014/

[7268015] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:35:06
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268015/

[7268016] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:28:38
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268016/

[7268018] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:09:35
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268018/

[7268019] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:56:51
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268019/

[7268020] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:22:45
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268020/

[7268022] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}}
time:   00:10:37
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268022/

[7268024] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   00:32:06
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268024/

[7268025] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:29:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268025/

[7268026] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:29:54
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268026/

[7268027] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:35:38
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268027/

[7268028] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:21:20
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268028/

[7268029] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}}
time:   00:26:47
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268029/

[7268030] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:22:35
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268030/

[7268031] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
time:   00:51:09
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268031/

[7268032] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:27:43
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268032/

[7268033] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:16:08
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268033/

[7268034] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:22:28
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268034/

[7268035] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:24:15
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268035/

[7268036] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:20:08
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268036/

[7268037] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:44:57
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268037/

[7268038] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:27:42
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268038/

[7268039] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}}
time:   00:38:47
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268039/

[7268041] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:36:40
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268041/

[7268042] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:34:00
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268042/

[7268044] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:55:38
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268044/

[7268045] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:00:44
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268045/

[7268046] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:16:02
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268046/

[7268047] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:32:04
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268047/

[7268048] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}}
time:   00:16:57
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268048/

[7268050] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:37:32
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268050/

[7268051] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}}
time:   00:34:22
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268051/

[7268052] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:20:58
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268052/

[7268053] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:27:07
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268053/

[7268054] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:23:30
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268054/

[7268055] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:20:33
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268055/

[7268056] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:25:01
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268056/

[7268057] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:16:51
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268057/

[7268058] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:24
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268058/

[7268059] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:16:44
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268059/

[7268060] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:19:38
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268060/

[7268061] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:32:07
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268061/

[7268063] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:22:57
info:   http://pulpito.ceph.com/vshankar-2023-05-09_07:29:33-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7268063/