2022-07-06T14:37:48.179 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3
2022-07-06T14:37:48.180 INFO:teuthology.results:subset: '1/32'
2022-07-06T14:37:48.181 INFO:teuthology.results:seed: '3567'
2022-07-06T17:16:34.400 INFO:root:teuthology version: 1.1.1.dev381+g10062088
2022-07-06T17:16:34.436 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-07-06T19:36:47.918 INFO:teuthology.results:Tests finished! gathering results...
2022-07-06T19:37:02.595 INFO:scrape:Found 144 jobs
2022-07-06T19:37:21.391 INFO:scrape:Found 12 distinct failure reasons
2022-07-06T19:37:21.392 INFO:scrape:Failure: {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}}
2022-07-06T19:37:21.393 INFO:scrape:5 jobs: ['6916960', '6916911', '6916849', '6916935', '6916843']
2022-07-06T19:37:21.393 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}']
2022-07-06T19:37:21.394 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-quota}', 'tasks/fsstress', 'tasks/{0-check-counter', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'validater/valgrind}', 'workunit/suites/blogbench}}', 'wsync/yes}']
2022-07-06T19:37:21.394 INFO:scrape:
2022-07-06T19:37:21.395 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi105.front.sepia.ceph.com: ['type=AVC msg=audit(1657127634.380:193): avc:  denied  { node_bind } for  pid=1436 comm="ping" saddr=172.21.15.105 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1']
2022-07-06T19:37:21.395 INFO:scrape:1 jobs: ['6916962']
2022-07-06T19:37:21.396 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/suites/ffsb}}', 'wsync/yes}']
2022-07-06T19:37:21.396 INFO:scrape:
2022-07-06T19:37:21.397 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2022-07-06T19:37:21.397 INFO:scrape:1 jobs: ['6916932']
2022-07-06T19:37:21.398 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}']
2022-07-06T19:37:21.398 INFO:scrape:
2022-07-06T19:37:21.399 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2022-07-06T19:37:21.399 INFO:scrape:5 jobs: ['6916971', '6916873', '6916928', '6916950', '6916929']
2022-07-06T19:37:21.400 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{frag']
2022-07-06T19:37:21.400 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated-and-quota}', 'subvolume/{with-namespace-isolated}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'workunit/suites/pjd}}', 'wsync/no}', 'wsync/yes}']
2022-07-06T19:37:21.401 INFO:scrape:
2022-07-06T19:37:21.401 INFO:scrape:Failure: "2022-07-06T16:54:39.920250+0000 osd.3 (osd.3) 38 : cluster [ERR] 5.1s0 deep-scrub : stat mismatch, got 0/2 objects, 0/0 clones, 0/2 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/2242 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
2022-07-06T19:37:21.402 INFO:scrape:1 jobs: ['6916916']
2022-07-06T19:37:21.402 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-07-06T19:37:21.403 INFO:scrape:
2022-07-06T19:37:21.403 INFO:scrape:Failure: Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)
2022-07-06T19:37:21.403 INFO:scrape:1 jobs: ['6916829']
2022-07-06T19:37:21.404 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2022-07-06T19:37:21.404 INFO:scrape:
2022-07-06T19:37:21.405 INFO:scrape:Failure: Command failed on smithi175 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3f8d7ffd9d86adba704c7df9d64f474f3906bc33 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75839988-fd44-11ec-842d-001a4aab830c -- ceph orch daemon add osd smithi175:vg_nvme/lv_4'
2022-07-06T19:37:21.405 INFO:scrape:2 jobs: ['6916870', '6916846']
2022-07-06T19:37:21.406 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter']
2022-07-06T19:37:21.406 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{no-subvolume}', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/suites/iogen}}', 'workunit/suites/pjd}}']
2022-07-06T19:37:21.407 INFO:scrape:
2022-07-06T19:37:21.407 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2022-07-06T19:37:21.408 INFO:scrape:3 jobs: ['6916936', '6916941', '6916892']
2022-07-06T19:37:21.408 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'osd}', 'overrides/{frag', 'races', 'ranks/1', 'session_timeout', 'thrashosds-health}']
2022-07-06T19:37:21.409 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-07-06T19:37:21.409 INFO:scrape:
2022-07-06T19:37:21.410 INFO:scrape:Failure: "2022-07-06T16:59:30.048134+0000 mds.b (mds.0) 38 : cluster [WRN] client.4749 isn't responding to mclientcaps(revoke), ino 0x10000009e51 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 302.303637 seconds ago" in cluster log
2022-07-06T19:37:21.410 INFO:scrape:1 jobs: ['6916900']
2022-07-06T19:37:21.411 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-07-06T19:37:21.411 INFO:scrape:
2022-07-06T19:37:21.412 INFO:scrape:Failure: error during scrub thrashing: Command failed on smithi084 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status'
2022-07-06T19:37:21.412 INFO:scrape:1 jobs: ['6916914']
2022-07-06T19:37:21.413 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{with-no-extra-options}', 'tasks/{0-check-counter', 'workunit/suites/ffsb}}', 'wsync/yes}']
2022-07-06T19:37:21.413 INFO:scrape:
2022-07-06T19:37:21.414 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2022-07-06T19:37:21.414 INFO:scrape:1 jobs: ['6916913']
2022-07-06T19:37:21.415 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2022-07-06T19:37:21.415 INFO:scrape:
2022-07-06T19:37:21.416 INFO:scrape:Failure: "2022-07-06T17:38:07.174742+0000 mon.a (mon.0) 5912 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log
2022-07-06T19:37:21.416 INFO:scrape:1 jobs: ['6916920']
2022-07-06T19:37:21.417 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-07-06T19:37:21.417 INFO:scrape:
2022-07-06T19:37:21.424 INFO:teuthology.results:Sending results to rishabhddave@gmail.com,ridave@redhat.com: Test Run: rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/
failed:  18
dead:    5
running: 0
waiting: 0
queued:  0
passed:  121


Fail
=================================================================
[6916829]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   02:06:29
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916829/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916829/
sentry: https://sentry.ceph.com/organizations/ceph/?query=04ac943752aa48d6a3902f92789f002c

    Test failure: test_mirroring_init_failure_with_recovery
    (tasks.cephfs.test_mirroring.TestMirroring)

[6916846]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:16:31
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916846/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916846/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e3241faeb454d98bc56cf83b05dc540

    Command failed on smithi044 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:3f8d7ffd9d86adba704c7df9d64f474f3906bc33 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d164c5fe-
    fd41-11ec-842d-001a4aab830c -- ceph orch daemon add osd
    smithi044:vg_nvme/lv_2'

[6916870]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:14:29
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916870/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916870/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e3bc63aa87ee45fe9c9699505be05d0b

    Command failed on smithi175 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:3f8d7ffd9d86adba704c7df9d64f474f3906bc33 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    75839988-fd44-11ec-842d-001a4aab830c -- ceph orch daemon add osd
    smithi175:vg_nvme/lv_4'

[6916873]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:24:29
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916873/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916873/
sentry: https://sentry.ceph.com/organizations/ceph/?query=36fcaea1f10043939ff98dce4ba4f6bb

    Command failed (workunit test suites/pjd.sh) on smithi039 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6916892]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:57:15
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916892/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916892/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6ea029236e214ed6b4d75daaf42063f9

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi003 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[6916900]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   02:56:32
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916900/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916900/

    "2022-07-06T16:59:30.048134+0000 mds.b (mds.0) 38 : cluster [WRN]
    client.4749 isn't responding to mclientcaps(revoke), ino 0x10000009e51
    pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 302.303637 seconds ago" in
    cluster log

[6916913]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:24:34
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916913/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916913/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bb7572aea0584e818abbd6484f1ade2b

    reached maximum tries (90) after waiting for 540 seconds

[6916914]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   01:21:06
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916914/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916914/

    error during scrub thrashing: Command failed on smithi084 with status 1:
    'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
    timeout 120 ceph --cluster ceph tell mds.1:0 scrub status'

[6916916]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:24:07
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916916/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916916/

    "2022-07-06T16:54:39.920250+0000 osd.3 (osd.3) 38 : cluster [ERR] 5.1s0
    deep-scrub : stat mismatch, got 0/2 objects, 0/0 clones, 0/2 dirty, 0/0
    omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/2242 bytes, 0/0
    manifest objects, 0/0 hit_set_archive bytes." in cluster log

[6916920]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   03:03:40
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916920/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916920/

    "2022-07-06T17:38:07.174742+0000 mon.a (mon.0) 5912 : cluster [WRN] Health
    check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log

[6916928]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:31:46
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916928/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916928/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6c0cf177ecbc44d39ab5dd3c5883a42b

    Command failed (workunit test suites/pjd.sh) on smithi081 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6916929]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:21:28
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916929/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916929/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ac4aeaf43f684ce7816a98d4c2fa25da

    Command failed (workunit test suites/pjd.sh) on smithi125 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6916932]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
-----------------------------------------------------------------
time:   00:14:32
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916932/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916932/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a5da09aee084faaba2eb151fd33d7ae

    Test failure: test_client_cache_size
    (tasks.cephfs.test_client_limits.TestClientLimits)

[6916936]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   01:36:46
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916936/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916936/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c19a9e6853784b928b5308f77644ae02

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi157 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[6916941]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:39:10
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916941/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916941/
sentry: https://sentry.ceph.com/organizations/ceph/?query=477dd2ba3a3a406dad14c1474f368015

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi019 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[6916950]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:31:40
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916950/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916950/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b1bc8d08524f4b14b13c784d8c3eeda1

    Command failed (workunit test suites/pjd.sh) on smithi029 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6916962]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   01:06:23
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916962/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916962/

    SELinux denials found on ubuntu@smithi105.front.sepia.ceph.com: ['type=AVC
    msg=audit(1657127634.380:193): avc:  denied  { node_bind } for  pid=1436
    comm="ping" saddr=172.21.15.105 scontext=system_u:system_r:ping_t:s0
    tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1']

[6916971]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:34:04
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916971/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916971/
sentry: https://sentry.ceph.com/organizations/ceph/?query=368279e7206d4a4fae71dcad7a6c99b8

    Command failed (workunit test suites/pjd.sh) on smithi081 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=3f8d7ffd9d86adba704c7df9d64f474f3906bc33
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'



Dead
=================================================================
[6916843]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:09:18
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916843/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916843/
sentry: https://sentry.ceph.com/organizations/ceph/?query=debbe4f2fbce49e9a6c64e297e3c9d1c

    {'smithi143.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6916849]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   00:09:40
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916849/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916849/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3e5db5f81ba54737aa1c87e02993ca9c

    {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6916911]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:09:16
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916911/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916911/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0611ac042c33405b9f9439e43a573065

    {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6916935]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:09:14
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916935/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916935/
sentry: https://sentry.ceph.com/organizations/ceph/?query=afa041edb28346be81329b0ba333d94b

    {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6916960]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:11:52
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916960/
log:    http://qa-proxy.ceph.com/teuthology/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916960/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bae7c634c26a49c7a3574c912131b29d

    {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}



Pass
=================================================================
[6916828] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:21:17
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916828/

[6916830] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/direct_io}}
time:   00:30:00
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916830/

[6916831] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/misc}}
time:   01:13:20
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916831/

[6916832] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:24:50
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916832/

[6916833] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:57:46
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916833/

[6916834] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:19:59
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916834/

[6916835] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:08
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916835/

[6916836] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:48:19
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916836/

[6916837] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/dbench}}
time:   01:59:22
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916837/

[6916838] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:20:09
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916838/

[6916839] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:24:23
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916839/

[6916840] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:22:54
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916840/

[6916841] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsx}}
time:   00:41:51
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916841/

[6916842] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:24:14
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916842/

[6916844] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:22:28
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916844/

[6916845] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:23:10
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916845/

[6916847] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:38
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916847/

[6916848] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:21:17
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916848/

[6916850] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/direct_io}}
time:   00:31:13
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916850/

[6916851] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:22:46
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916851/

[6916852] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:56:13
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916852/

[6916853] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:22:03
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916853/

[6916854] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:40:29
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916854/

[6916855] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:14:58
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916855/

[6916856] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:17:30
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916856/

[6916857] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/dbench}}
time:   02:01:00
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916857/

[6916858] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:22:35
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916858/

[6916859] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:25:19
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916859/

[6916860] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:23:38
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916860/

[6916861] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:33:20
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916861/

[6916862] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:14:42
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916862/

[6916863] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:22:55
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916863/

[6916864] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsx}}
time:   00:40:53
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916864/

[6916865] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   02:08:04
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916865/

[6916866] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:19:18
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916866/

[6916867] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:13:21
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916867/

[6916868] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:38:51
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916868/

[6916869] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:20:53
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916869/

[6916871] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:21:10
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916871/

[6916872] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:31:54
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916872/

[6916874] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   00:26:04
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916874/

[6916875] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:19:33
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916875/

[6916876] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:50:57
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916876/

[6916877] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   01:14:47
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916877/

[6916878] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:53:33
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916878/

[6916879] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:25:56
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916879/

[6916880] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:32:51
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916880/

[6916881] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:19:04
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916881/

[6916882] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:53:57
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916882/

[6916883] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:22:19
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916883/

[6916884] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:29:32
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916884/

[6916885] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/ffsb}}
time:   01:08:04
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916885/

[6916886] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:04
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916886/

[6916887] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:22:42
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916887/

[6916888] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/norstats}}
time:   00:26:22
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916888/

[6916889] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:39:01
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916889/

[6916890] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:22:45
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916890/

[6916891] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:20:27
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916891/

[6916893] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:44:25
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916893/

[6916894] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:23:17
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916894/

[6916895] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:30:54
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916895/

[6916896] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/iozone}}
time:   00:28:41
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916896/

[6916897] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:13:27
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916897/

[6916898] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:21:30
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916898/

[6916899] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:20
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916899/

[6916901] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/pjd}}
time:   00:31:36
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916901/

[6916902] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:37:26
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916902/

[6916903] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:23:21
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916903/

[6916904] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:44
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916904/

[6916905] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:31:07
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916905/

[6916906] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
time:   01:41:39
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916906/

[6916907] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/misc}}
time:   01:06:55
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916907/

[6916908] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:19:05
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916908/

[6916909] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:33:07
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916909/

[6916910] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:23:28
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916910/

[6916912] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:04:53
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916912/

[6916915] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:12:50
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916915/

[6916917] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:34:14
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916917/

[6916918] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:21:03
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916918/

[6916919] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:26:56
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916919/

[6916921] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:59
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916921/

[6916922] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:30:24
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916922/

[6916923] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:23:40
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916923/

[6916924] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:21:13
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916924/

[6916925] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:32:39
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916925/

[6916926] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:52:24
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916926/

[6916927] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:23:17
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916927/

[6916930] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:48:49
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916930/

[6916931] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:33:05
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916931/

[6916933] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:30:18
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916933/

[6916934] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:18:59
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916934/

[6916937] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:20:52
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916937/

[6916938] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:39:31
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916938/

[6916939] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/ffsb}}
time:   00:39:50
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916939/

[6916940] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:20:34
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916940/

[6916942] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:24:25
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916942/

[6916943] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:33:11
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916943/

[6916944] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:22:09
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916944/

[6916945] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:33:34
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916945/

[6916946] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:19:31
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916946/

[6916947] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:21:40
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916947/

[6916948] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:19:04
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916948/

[6916949] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:08:33
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916949/

[6916951] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:52:40
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916951/

[6916952] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:20:05
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916952/

[6916953] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:33:35
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916953/

[6916954] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:30:00
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916954/

[6916955] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
time:   01:32:57
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916955/

[6916956] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:36:36
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916956/

[6916957] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:18
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916957/

[6916958] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:21:03
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916958/

[6916959] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:20:23
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916959/

[6916961] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:21:41
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916961/

[6916963] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:22:17
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916963/

[6916964] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:26:16
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916964/

[6916965] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:34:19
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916965/

[6916966] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:35:34
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916966/

[6916967] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:20:52
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916967/

[6916968] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:20:44
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916968/

[6916969] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:23:24
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916969/

[6916970] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:21:43
info:   http://pulpito.ceph.com/rishabh-2022-07-06_05:27:56-fs-wip-rishabh-qa-cleanup-distro-default-smithi/6916970/