2022-03-03T09:45:30.961 INFO:root:teuthology version: 1.1.0-eeaeee38
2022-03-03T09:45:30.963 INFO:teuthology.results:subset: '25/32'
2022-03-03T09:45:30.963 INFO:teuthology.results:seed: '5082'
2022-03-03T10:25:47.374 INFO:root:teuthology version: 1.1.0-eeaeee38
2022-03-03T10:25:47.437 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-03-03T17:07:18.490 INFO:teuthology.results:Tests finished! gathering results...
2022-03-03T17:07:24.799 INFO:scrape:Found 68 jobs
2022-03-03T17:07:43.015 INFO:scrape:Found 14 distinct failure reasons
2022-03-03T17:07:43.016 INFO:scrape:Failure: Test failure: test_unmount_for_evicted_client (tasks.cephfs.test_client_recovery.TestClientRecovery)
2022-03-03T17:07:43.017 INFO:scrape:1 jobs: ['6717811']
2022-03-03T17:07:43.017 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/client-recovery}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.018 INFO:scrape:
2022-03-03T17:07:43.018 INFO:scrape:Failure: Command failed (workunit test fs/misc/rstats.sh) on smithi084 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e036ead120ca31fe765e9c3c926bd809b247210f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/rstats.sh'
2022-03-03T17:07:43.019 INFO:scrape:1 jobs: ['6717812']
2022-03-03T17:07:43.019 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/3', 'rhel_8}', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{no}}']
2022-03-03T17:07:43.020 INFO:scrape:
2022-03-03T17:07:43.020 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
2022-03-03T17:07:43.021 INFO:scrape:3 jobs: ['6717844', '6717834', '6717802']
2022-03-03T17:07:43.021 INFO:scrape:suites intersection: ['1-ranks/2', '1-volume/{0-create', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.022 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.022 INFO:scrape:
2022-03-03T17:07:43.023 INFO:scrape:Dead: b'2022-03-03T16:59:56.802 DEBUG:teuthology.exit:Finished running handlers'
2022-03-03T17:07:43.023 INFO:scrape:2 jobs: ['6717853', '6717848']
2022-03-03T17:07:43.024 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.024 INFO:scrape:suites union: ['1-cephadm', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{pg-warn', 'ranks/3', 'rhel_8}', 'roles', 'scrub/no', 'session_timeout', 'standby-replay', 'syntax', 'tasks/{0-check-counter', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}', 'wsync/{no}}']
2022-03-03T17:07:43.025 INFO:scrape:
2022-03-03T17:07:43.025 INFO:scrape:Timeout 5h running clone.client.0/qa/workunits/suites/ffsb.sh
2022-03-03T17:07:43.025 INFO:scrape:1 jobs: ['6717818']
2022-03-03T17:07:43.026 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}', 'wsync/{yes}}']
2022-03-03T17:07:43.026 INFO:scrape:
2022-03-03T17:07:43.027 INFO:scrape:Failure: "2022-03-03T10:13:16.128605+0000 mon.a (mon.0) 356 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022-03-03T17:07:43.027 INFO:scrape:2 jobs: ['6717791', '6717843']
2022-03-03T17:07:43.028 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'osd}', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.028 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.029 INFO:scrape:
2022-03-03T17:07:43.029 INFO:scrape:Failure: "2022-03-03T10:57:46.385365+0000 mds.a (mds.0) 27 : cluster [WRN] client.4798 isn't responding to mclientcaps(revoke), ino 0x10000003ea3 pending pAsLsXsFscr issued pAsLsXsFscrb, sent 302.236048 seconds ago" in cluster log
2022-03-03T17:07:43.030 INFO:scrape:2 jobs: ['6717815', '6717841']
2022-03-03T17:07:43.030 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.031 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.031 INFO:scrape:
2022-03-03T17:07:43.032 INFO:scrape:Failure: Command failed on smithi062 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'
2022-03-03T17:07:43.032 INFO:scrape:2 jobs: ['6717846', '6717800']
2022-03-03T17:07:43.033 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{yes}}']
2022-03-03T17:07:43.033 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{yes}}']
2022-03-03T17:07:43.034 INFO:scrape:
2022-03-03T17:07:43.034 INFO:scrape:Failure: "2022-03-03T14:25:22.836889+0000 osd.1 (osd.1) 24 : cluster [ERR] 5.9s0 deep-scrub : stat mismatch, got 14/13 objects, 0/0 clones, 14/13 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 12158/11954 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
2022-03-03T17:07:43.035 INFO:scrape:1 jobs: ['6717822']
2022-03-03T17:07:43.035 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.036 INFO:scrape:
2022-03-03T17:07:43.036 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2022-03-03T17:07:43.037 INFO:scrape:1 jobs: ['6717794']
2022-03-03T17:07:43.037 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2022-03-03T17:07:43.038 INFO:scrape:
2022-03-03T17:07:43.038 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh
2022-03-03T17:07:43.039 INFO:scrape:1 jobs: ['6717833']
2022-03-03T17:07:43.039 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.040 INFO:scrape:
2022-03-03T17:07:43.040 INFO:scrape:Failure: Test failure: test_fuse_mount_on_already_mounted_path (tasks.cephfs.test_misc.TestMisc)
2022-03-03T17:07:43.041 INFO:scrape:2 jobs: ['6717793', '6717824']
2022-03-03T17:07:43.041 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{whitelist_health', 'tasks/cephfs_misc_tests}', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.042 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{whitelist_health', 'tasks/cephfs_misc_tests}', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.042 INFO:scrape:
2022-03-03T17:07:43.042 INFO:scrape:Crash: Found coredumps on ubuntu@smithi112.front.sepia.ceph.com
2022-03-03T17:07:43.043 INFO:scrape:ceph version 17.0.0-10900-gc7225ee0 (c7225ee0e6a2b75adcf6781e173f5203343288da) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f8486e343c0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7f84870f8834]
 5: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f84870f89de]
 6: (MDCache::dispatch_fragment_dir(boost::intrusive_ptr<MDRequestImpl>&)+0x1701) [0x55674d35d4c1]
 7: (MDCache::fragment_frozen(boost::intrusive_ptr<MDRequestImpl>&, int)+0xf3) [0x55674d35d683]
 8: (MDCache::fragment_mark_and_complete(boost::intrusive_ptr<MDRequestImpl>&)+0xe16) [0x55674d35e6a6]
 9: (MDCache::merge_dir(CInode*, frag_t)+0x496) [0x55674d35f5a6]
 10: ceph-mds(+0x451659) [0x55674d466659]
 11: (Context::complete(int)+0xd) [0x55674d1e8efd]
 12: (CommonSafeTimer<ceph::fair_mutex>::timer_thread()+0x15d) [0x7f848720531d]
 13: (CommonSafeTimerThread<ceph::fair_mutex>::entry()+0x11) [0x7f8487206821]
 14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f8486e28609]
 15: clone()
2022-03-03T17:07:43.043 INFO:scrape:1 jobs: ['6717806']
2022-03-03T17:07:43.044 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.044 INFO:scrape:
2022-03-03T17:07:43.045 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2022-03-03T17:07:43.045 INFO:scrape:1 jobs: ['6717836']
2022-03-03T17:07:43.046 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/pool-perm}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-03-03T17:07:43.046 INFO:scrape:
2022-03-03T17:07:43.050 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/
failed:  18
dead:    3
running: 0
waiting: 0
queued:  0
passed:  47


Fail
=================================================================
[6717791]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:38:19
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717791/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717791/

    "2022-03-03T10:13:16.128605+0000 mon.a (mon.0) 356 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[6717793]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:22:04
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717793/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717793/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0193d5fdc29b4f33ad446ba9f9b33576

    Test failure: test_fuse_mount_on_already_mounted_path
    (tasks.cephfs.test_misc.TestMisc)

[6717794]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   01:33:47
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717794/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717794/
sentry: https://sentry.ceph.com/organizations/ceph/?query=71c7b6c58d954feb9fe8df1c2867f905

    Test failure: test_cephfs_mirror_restart_sync_on_blocklist
    (tasks.cephfs.test_mirroring.TestMirroring)

[6717800]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
-----------------------------------------------------------------
time:   00:55:28
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717800/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717800/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3c29d4844c4e479d8c5a83e057734272

    Command failed on smithi133 with status 1: 'sudo rm -rf --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp'

[6717806]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   01:43:55
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717806/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717806/

    Found coredumps on ubuntu@smithi112.front.sepia.ceph.com

[6717811]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:37:26
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717811/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717811/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3fce52ac34024f15b75c516c573b8244

    Test failure: test_unmount_for_evicted_client
    (tasks.cephfs.test_client_recovery.TestClientRecovery)

[6717812]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}}
-----------------------------------------------------------------
time:   00:44:38
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717812/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717812/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dbed6663ea7f43fcb1d62de6b7391eed

    Command failed (workunit test fs/misc/rstats.sh) on smithi084 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e036ead120ca31fe765e9c3c926bd809b247210f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/rstats.sh'

[6717815]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   03:26:37
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717815/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717815/

    "2022-03-03T10:57:46.385365+0000 mds.a (mds.0) 27 : cluster [WRN]
    client.4798 isn't responding to mclientcaps(revoke), ino 0x10000003ea3
    pending pAsLsXsFscr issued pAsLsXsFscrb, sent 302.236048 seconds ago" in
    cluster log

[6717818]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
-----------------------------------------------------------------
time:   05:31:19
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717818/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717818/
sentry: https://sentry.ceph.com/organizations/ceph/?query=72ce2dc8285247368bacd5d0004d9455

    Command failed (workunit test suites/ffsb.sh) on smithi045 with status 124:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e036ead120ca31fe765e9c3c926bd809b247210f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 5h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

[6717822]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   04:43:04
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717822/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717822/

    "2022-03-03T14:25:22.836889+0000 osd.1 (osd.1) 24 : cluster [ERR] 5.9s0
    deep-scrub : stat mismatch, got 14/13 objects, 0/0 clones, 14/13 dirty, 0/0
    omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 12158/11954 bytes,
    0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log

[6717824]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:22:15
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717824/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717824/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fd7ff2820939409fb813a32b85f16826

    Test failure: test_fuse_mount_on_already_mounted_path
    (tasks.cephfs.test_misc.TestMisc)

[6717833]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   05:24:14
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717833/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717833/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a30a01d2660f4cec9c49da3beb8b6b9a

    Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi025 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e036ead120ca31fe765e9c3c926bd809b247210f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh'

[6717834]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   03:20:58
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717834/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717834/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4ded2b82bb56489aa04934e3211fe4c4

    Command failed (workunit test suites/fsstress.sh) on smithi181 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e036ead120ca31fe765e9c3c926bd809b247210f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[6717836]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm}
-----------------------------------------------------------------
time:   00:14:07
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717836/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717836/
sentry: https://sentry.ceph.com/organizations/ceph/?query=451a6c2d61e34982a4833bde75ae56c6

    Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

[6717841]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   05:02:11
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717841/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717841/

    "2022-03-03T11:39:32.721028+0000 mds.f (mds.0) 29 : cluster [WRN]
    client.4783 isn't responding to mclientcaps(revoke), ino 0x100000034b5
    pending pAsLsXsFscr issued pAsLsXsFscrb, sent 300.006970 seconds ago" in
    cluster log

[6717843]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:31:06
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717843/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717843/

    "2022-03-03T10:35:03.686385+0000 mon.a (mon.0) 323 : cluster [WRN] Health
    check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY)"
    in cluster log

[6717844]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   03:19:39
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717844/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717844/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5873d485fdfc423aa8e8b2b2da009231

    Command failed (workunit test suites/fsstress.sh) on smithi170 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e036ead120ca31fe765e9c3c926bd809b247210f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[6717846]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
-----------------------------------------------------------------
time:   00:55:57
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717846/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717846/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7473bf7d3ac04d65b05698f1dd98b8f8

    Command failed on smithi062 with status 1: 'sudo rm -rf --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp'



Dead
=================================================================
[6717802]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   03:35:49
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717802/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717802/
sentry: https://sentry.ceph.com/organizations/ceph/?query=62e4399029594e0a9e60198a98ced50c

    Command failed (workunit test suites/fsstress.sh) on smithi089 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=e036ead120ca31fe765e9c3c926bd809b247210f
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[6717848]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717848/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717848/

    hit max job timeout

[6717853]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717853/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717853/

    hit max job timeout



Pass
=================================================================
[6717788] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:18:35
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717788/

[6717789] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:14:37
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717789/

[6717790] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
time:   00:31:45
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717790/

[6717792] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:19:00
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717792/

[6717795] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:19:20
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717795/

[6717796] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:16:57
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717796/

[6717797] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:18:35
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717797/

[6717798] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:18:52
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717798/

[6717799] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:53:43
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717799/

[6717801] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:20:21
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717801/

[6717803] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:16:55
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717803/

[6717804] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   00:28:17
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717804/

[6717805] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:17:27
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717805/

[6717807] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:17:52
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717807/

[6717808] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:17:58
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717808/

[6717809] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
time:   00:30:48
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717809/

[6717810] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:17:47
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717810/

[6717813] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:18:24
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717813/

[6717814] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:16:48
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717814/

[6717816] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:59:03
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717816/

[6717817] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:16:39
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717817/

[6717819] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
time:   00:33:31
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717819/

[6717820] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:45
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717820/

[6717821] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:16:16
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717821/

[6717823] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:21:39
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717823/

[6717825] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:31:43
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717825/

[6717826] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}}
time:   00:44:35
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717826/

[6717827] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:16:20
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717827/

[6717828] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}}
time:   00:27:19
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717828/

[6717829] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:53
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717829/

[6717830] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:18:11
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717830/

[6717831] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:18:53
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717831/

[6717832] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:17:07
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717832/

[6717835] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:17:10
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717835/

[6717837] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   00:31:27
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717837/

[6717838] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:43
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717838/

[6717839] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:52:58
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717839/

[6717840] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:18:32
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717840/

[6717842] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:16:14
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717842/

[6717845] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:17:30
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717845/

[6717847] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:17:23
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717847/

[6717849] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:16:28
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717849/

[6717850] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   00:31:27
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717850/

[6717851] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:18:13
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717851/

[6717852] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:18:08
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717852/

[6717854] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:17:03
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717854/

[6717855] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
time:   00:45:09
info:   http://pulpito.ceph.com/vshankar-2022-03-03_09:44:37-fs-master-testing-default-smithi/6717855/