2022-10-20T06:06:18.172 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-20T06:06:18.174 INFO:teuthology.results:subset: '69/256'
2022-10-20T06:06:18.174 INFO:teuthology.results:seed: '5160'
2022-10-20T07:30:42.255 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-20T07:30:42.281 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-10-20T18:51:34.594 INFO:teuthology.results:Tests finished! gathering results...
2022-10-20T18:51:39.750 INFO:scrape:Found 53 jobs
2022-10-20T18:51:44.718 INFO:scrape:Found 7 distinct failure reasons
2022-10-20T18:51:44.719 INFO:scrape:Failure: Command failed on smithi130 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1eb91db4-5044-11ed-8437-001a4aab830c -e sha1=b4d1a1999fef7ac95183f3533a3f0ab581066286 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"
2022-10-20T18:51:44.720 INFO:scrape:6 jobs: ['7075261', '7075276', '7075288', '7075299', '7075254', '7075280']
2022-10-20T18:51:44.720 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4']
2022-10-20T18:51:44.721 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4']
2022-10-20T18:51:44.721 INFO:scrape:
2022-10-20T18:51:44.722 INFO:scrape:Failure: Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace)
2022-10-20T18:51:44.722 INFO:scrape:1 jobs: ['7075270']
2022-10-20T18:51:44.723 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/backtrace}']
2022-10-20T18:51:44.723 INFO:scrape:
2022-10-20T18:51:44.723 INFO:scrape:Failure: "1666252962.0130157 mon.a (mon.0) 1485 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
2022-10-20T18:51:44.724 INFO:scrape:2 jobs: ['7075294', '7075282']
2022-10-20T18:51:44.724 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{rhel_8}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2022-10-20T18:51:44.725 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/failover}', 'tasks/snap-schedule}']
2022-10-20T18:51:44.725 INFO:scrape:
2022-10-20T18:51:44.726 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6fb859258928a5d76fd5f6b6802b10593037f421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2022-10-20T18:51:44.726 INFO:scrape:1 jobs: ['7075291']
2022-10-20T18:51:44.726 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}']
2022-10-20T18:51:44.727 INFO:scrape:
2022-10-20T18:51:44.727 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi189 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6fb859258928a5d76fd5f6b6802b10593037f421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2022-10-20T18:51:44.728 INFO:scrape:1 jobs: ['7075269']
2022-10-20T18:51:44.728 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2022-10-20T18:51:44.729 INFO:scrape:
2022-10-20T18:51:44.729 INFO:scrape:Failure: "1666268906.5773723 osd.4 (osd.4) 50 : cluster [ERR] 5.1ds0 deep-scrub : stat mismatch, got 1/13 objects, 0/0 clones, 1/13 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1314/19815 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
2022-10-20T18:51:44.729 INFO:scrape:1 jobs: ['7075298']
2022-10-20T18:51:44.730 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-10-20T18:51:44.730 INFO:scrape:
2022-10-20T18:51:44.731 INFO:scrape:Dead: b'2022-10-20T18:45:29.207 DEBUG:teuthology.exit:Finished running handlers'
2022-10-20T18:51:44.731 INFO:scrape:1 jobs: ['7075268']
2022-10-20T18:51:44.732 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2022-10-20T18:51:44.732 INFO:scrape:
2022-10-20T18:51:44.735 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/
failed:  12
dead:    1
running: 0
waiting: 0
queued:  0
passed:  40


Fail
=================================================================
[7075254]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:22:42
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075254/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075254/
sentry: https://sentry.ceph.com/organizations/ceph/?query=57a52f85ee4241799f44862e52d77c6c

    Command failed on smithi035 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    14cbaa4e-5042-11ed-8437-001a4aab830c -e
    sha1=b4d1a1999fef7ac95183f3533a3f0ab581066286 -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

[7075261]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:22:41
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075261/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075261/
sentry: https://sentry.ceph.com/organizations/ceph/?query=82232aed44464942bbb6f61c61eebb87

    Command failed on smithi130 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    1eb91db4-5044-11ed-8437-001a4aab830c -e
    sha1=b4d1a1999fef7ac95183f3533a3f0ab581066286 -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

[7075269]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:28:14
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075269/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075269/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4379aeae98a7455fa8de69e1dfd6beb4

    Command failed (workunit test kernel_untar_build.sh) on smithi189 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6fb859258928a5d76fd5f6b6802b10593037f421
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7075270]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
-----------------------------------------------------------------
time:   00:15:44
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075270/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075270/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b73134e802384465b287a92cdfd13409

    Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace)

[7075276]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:23:10
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075276/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075276/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6501a5b89516459db403f1fbde35c906

    Command failed on smithi035 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    afb6d48a-5046-11ed-8437-001a4aab830c -e
    sha1=b4d1a1999fef7ac95183f3533a3f0ab581066286 -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

[7075280]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:22:16
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075280/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075280/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b0a0abb89c4848e9bcb5473ab3d47184

    Command failed on smithi134 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    22f54e18-5047-11ed-8437-001a4aab830c -e
    sha1=b4d1a1999fef7ac95183f3533a3f0ab581066286 -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

[7075282]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:57:27
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075282/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075282/

    "1666251709.760383 mon.a (mon.0) 2677 : cluster [WRN] Health check failed:
    Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

[7075288]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:22:39
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075288/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075288/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d05d038d7d624156b47d30a3cf4086c7

    Command failed on smithi130 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    bf121816-5048-11ed-8437-001a4aab830c -e
    sha1=b4d1a1999fef7ac95183f3533a3f0ab581066286 -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

[7075291]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:55:38
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075291/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075291/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2bc2a8db74e14baabf2919a7db565d7e

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi043 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6fb859258928a5d76fd5f6b6802b10593037f421
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7075294]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:58:07
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075294/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075294/

    "1666252962.0130157 mon.a (mon.0) 1485 : cluster [WRN] Health check failed:
    Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log

[7075298]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   05:23:17
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075298/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075298/

    "1666268906.5773723 osd.4 (osd.4) 50 : cluster [ERR] 5.1ds0 deep-scrub :
    stat mismatch, got 1/13 objects, 0/0 clones, 1/13 dirty, 0/0 omap, 0/0
    pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1314/19815 bytes, 0/0 manifest
    objects, 0/0 hit_set_archive bytes." in cluster log

[7075299]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:22:41
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075299/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075299/
sentry: https://sentry.ceph.com/organizations/ceph/?query=27bd7e06431e481aa84e731ec1dc6c32

    Command failed on smithi074 with status 22: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    2195aa68-504c-11ed-8437-001a4aab830c -e
    sha1=b4d1a1999fef7ac95183f3533a3f0ab581066286 -- bash -c 'ceph orch upgrade
    start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"



Dead
=================================================================
[7075268]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075268/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075268/

    hit max job timeout



Pass
=================================================================
[7075249] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:22:05
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075249/

[7075250] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:22:42
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075250/

[7075251] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   03:27:46
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075251/

[7075252] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:19:51
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075252/

[7075253] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:35:21
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075253/

[7075255] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:21:12
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075255/

[7075256] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   01:07:27
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075256/

[7075257] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:12:09
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075257/

[7075258] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:20:22
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075258/

[7075259] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:27:11
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075259/

[7075260] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:19:36
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075260/

[7075262] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:32:34
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075262/

[7075263] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:19:49
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075263/

[7075264] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror}
time:   02:18:49
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075264/

[7075265] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:54:25
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075265/

[7075266] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:21:34
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075266/

[7075267] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/fstop}
time:   00:18:34
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075267/

[7075271] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:17:19
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075271/

[7075272] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:25:30
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075272/

[7075273] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:22:47
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075273/

[7075274] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:46:02
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075274/

[7075275] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   00:37:08
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075275/

[7075277] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:55:11
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075277/

[7075278] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:33:28
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075278/

[7075279] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:26:38
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075279/

[7075281] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:25:27
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075281/

[7075283] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   02:10:04
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075283/

[7075284] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:21:10
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075284/

[7075285] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   01:00:18
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075285/

[7075286] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:54:04
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075286/

[7075287] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:21:39
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075287/

[7075289] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:25:36
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075289/

[7075290] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:19:49
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075290/

[7075292] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:12:34
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075292/

[7075293] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:23:00
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075293/

[7075295] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:24:28
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075295/

[7075296] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:21:52
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075296/

[7075297] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:43:04
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075297/

[7075300] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:34:44
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075300/

[7075301] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:19:01
info:   http://pulpito.ceph.com/vshankar-2022-10-20_06:04:44-fs-main-testing-default-smithi/7075301/