2022-09-27T03:15:18.780 INFO:root:teuthology version: 1.1.1.dev424+ga773b061
2022-09-27T03:15:18.781 INFO:teuthology.results:subset: '21/32'
2022-09-27T03:15:18.782 INFO:teuthology.results:seed: '9436'
2022-09-27T13:39:55.561 INFO:root:teuthology version: 1.1.1.dev424+ga773b061
2022-09-27T13:39:55.583 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-09-28T01:40:01.942 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2022-09-28T01:40:30.760 INFO:scrape:Found 293 jobs
2022-09-28T01:40:47.482 INFO:scrape:Found 12 distinct failure reasons
2022-09-28T01:40:47.484 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
2022-09-28T01:40:47.485 INFO:scrape:2 jobs: ['7044622', '7044754']
2022-09-28T01:40:47.486 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{legacy}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}']
2022-09-28T01:40:47.487 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}', 'wsync/{no}}', 'wsync/{yes}}']
2022-09-28T01:40:47.487 INFO:scrape:
2022-09-28T01:40:47.488 INFO:scrape:Failure: "2022-09-27T03:46:11.531996+0000 mon.a (mon.0) 1718 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
2022-09-28T01:40:47.488 INFO:scrape:3 jobs: ['7044589', '7044746', '7044817']
2022-09-28T01:40:47.489 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.489 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'overrides/{whitelist_health', 'rhel_8}', 'tasks/cephfs_misc_tests}', 'tasks/metrics}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.490 INFO:scrape:
2022-09-28T01:40:47.490 INFO:scrape:Failure: Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
2022-09-28T01:40:47.491 INFO:scrape:1 jobs: ['7044702']
2022-09-28T01:40:47.491 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.492 INFO:scrape:
2022-09-28T01:40:47.492 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2022-09-28T01:40:47.493 INFO:scrape:7 jobs: ['7044665', '7044736', '7044826', '7044777', '7044843', '7044700', '7044830']
2022-09-28T01:40:47.493 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.494 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{yes}}']
2022-09-28T01:40:47.494 INFO:scrape:
2022-09-28T01:40:47.495 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2022-09-28T01:40:47.495 INFO:scrape:1 jobs: ['7044752']
2022-09-28T01:40:47.495 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.496 INFO:scrape:
2022-09-28T01:40:47.496 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7e76448beefff9399729adb1dc5583efa7528947
2022-09-28T01:40:47.497 INFO:scrape:4 jobs: ['7044836', '7044785', '7044682', '7044570']
2022-09-28T01:40:47.497 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd}', 'overrides/{osd-asserts', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.498 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{osd-asserts', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.498 INFO:scrape:
2022-09-28T01:40:47.499 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi097 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7e76448beefff9399729adb1dc5583efa7528947 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2022-09-28T01:40:47.499 INFO:scrape:2 jobs: ['7044783', '7044749']
2022-09-28T01:40:47.500 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.500 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{whitelist_health', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.501 INFO:scrape:
2022-09-28T01:40:47.501 INFO:scrape:Failure: error during scrub thrashing: Command failed on smithi007 with status 13: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status'
2022-09-28T01:40:47.502 INFO:scrape:1 jobs: ['7044705']
2022-09-28T01:40:47.502 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{no}}']
2022-09-28T01:40:47.503 INFO:scrape:
2022-09-28T01:40:47.503 INFO:scrape:Failure: "2022-09-27T14:13:35.735336+0000 mds.c (mds.0) 6 : cluster [WRN] client.4657 isn't responding to mclientcaps(revoke), ino 0x1000000f589 pending pAsLsXsFsc issued pAsLsXsFscb, sent 304.859199 seconds ago" in cluster log
2022-09-28T01:40:47.503 INFO:scrape:3 jobs: ['7044850', '7044659', '7044585']
2022-09-28T01:40:47.504 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.505 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.505 INFO:scrape:
2022-09-28T01:40:47.506 INFO:scrape:Dead: b'2022-09-28T01:33:04.985 DEBUG:teuthology.exit:Finished running handlers'
2022-09-28T01:40:47.507 INFO:scrape:2 jobs: ['7044847', '7044839']
2022-09-28T01:40:47.507 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-09-28T01:40:47.507 INFO:scrape:suites union: ['1-cephadm', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-logrotate}', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{pg-warn', 'ranks/3', 'roles', 'scrub/yes', 'session_timeout', 'standby-replay', 'syntax', 'tasks/{0-check-counter', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}', 'wsync/{no}}']
2022-09-28T01:40:47.508 INFO:scrape:
2022-09-28T01:40:47.508 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2022-09-28T01:40:47.509 INFO:scrape:1 jobs: ['7044606']
2022-09-28T01:40:47.509 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2022-09-28T01:40:47.510 INFO:scrape:
2022-09-28T01:40:47.510 INFO:scrape:Failure: Command failed on smithi097 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:7e76448beefff9399729adb1dc5583efa7528947 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4e7fca0c-3e6a-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi097:vg_nvme/lv_3'
2022-09-28T01:40:47.511 INFO:scrape:1 jobs: ['7044852']
2022-09-28T01:40:47.511 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/1', 'rhel_8}', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}', 'wsync/{no}}']
2022-09-28T01:40:47.512 INFO:scrape:
2022-09-28T01:40:47.525 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/
failed:  26
dead:    1
running: 1
waiting: 0
queued:  0
passed:  265


Fail
=================================================================
[7044570]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:05:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044570/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044570/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13115765bc274794afd20c1a9a7021bf

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1
    =7e76448beefff9399729adb1dc5583efa7528947

[7044585]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   03:26:04
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044585/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044585/

    "2022-09-27T04:17:28.858892+0000 mds.d (mds.0) 1 : cluster [WRN]
    client.4692 isn't responding to mclientcaps(revoke), ino 0x10000003dd8
    pending pAsxLsXsxFscr issued pAsxLsXsxFsxcrwb, sent 300.005193 seconds ago"
    in cluster log

[7044589]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics}
-----------------------------------------------------------------
time:   00:25:03
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044589/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044589/

    "2022-09-27T03:46:11.531996+0000 mon.a (mon.0) 1718 : cluster [WRN] Health
    check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)"
    in cluster log

[7044606]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   01:30:09
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044606/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044606/
sentry: https://sentry.ceph.com/organizations/ceph/?query=78b953db53f44f24b01ec3fafe59b587

    Test failure: test_cephfs_mirror_restart_sync_on_blocklist
    (tasks.cephfs.test_mirroring.TestMirroring)

[7044622]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}}
-----------------------------------------------------------------
time:   01:03:39
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044622/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044622/

    The following counters failed to be set on mds daemons: {'mds.exported',
    'mds.imported'}

[7044659]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   03:50:22
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044659/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044659/

    "2022-09-27T10:53:44.711887+0000 mds.d (mds.1) 1 : cluster [WRN]
    client.4777 isn't responding to mclientcaps(revoke), ino 0x20000000334
    pending pFc issued pFcb, sent 300.005260 seconds ago" in cluster log

[7044665]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:18:00
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044665/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044665/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4191cc2a64f1440286e4aa016bab910e

    reached maximum tries (90) after waiting for 540 seconds

[7044682]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:05:50
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044682/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044682/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f4ba9079339045d1b6d827aaaa066b64

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1
    =7e76448beefff9399729adb1dc5583efa7528947

[7044700]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:19:19
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044700/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044700/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d0bf89ed98214299a9eb863c4c914204

    reached maximum tries (90) after waiting for 540 seconds

[7044702]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:28:42
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044702/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044702/
sentry: https://sentry.ceph.com/organizations/ceph/?query=58669e68fd4741b890c2acf2960e3f0c

    Test failure: test_subvolume_group_ls_filter_internal_directories
    (tasks.cephfs.test_volumes.TestSubvolumeGroups)

[7044705]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}}
-----------------------------------------------------------------
time:   00:56:41
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044705/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044705/

    error during scrub thrashing: Command failed on smithi007 with status 13:
    'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
    timeout 120 ceph --cluster ceph tell mds.1:0 scrub status'

[7044736]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:18:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044736/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044736/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3de3f9b259a5408abcf5625c4b3fe077

    reached maximum tries (90) after waiting for 540 seconds

[7044746]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:47:11
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044746/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044746/

    "2022-09-27T11:37:55.362288+0000 mon.a (mon.0) 921 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7044749]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:21:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044749/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044749/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b2f32c1264664ecf920bd79009ea6eb5

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi027 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=7e76448beefff9399729adb1dc5583efa7528947
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7044752]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:46:36
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044752/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044752/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9028aee0900e4bd083cb76b7c8b077cd

    Test failure: test_subvolume_snapshot_info_if_orphan_clone
    (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)

[7044754]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}}
-----------------------------------------------------------------
time:   01:15:57
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044754/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044754/

    The following counters failed to be set on mds daemons: {'mds.exported',
    'mds.imported'}

[7044777]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:19:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044777/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044777/
sentry: https://sentry.ceph.com/organizations/ceph/?query=60928a9f6f89485a9270a5c41287248f

    reached maximum tries (90) after waiting for 540 seconds

[7044783]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:30:58
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044783/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044783/
sentry: https://sentry.ceph.com/organizations/ceph/?query=80170278f128473190430413e5de3101

    Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on
    smithi097 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=7e76448beefff9399729adb1dc5583efa7528947
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-
    multiple-capsnaps.sh'

[7044785]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:06:00
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044785/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044785/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eff0c99a2e5b4473b586615c56e9d80d

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1
    =7e76448beefff9399729adb1dc5583efa7528947

[7044817]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics}
-----------------------------------------------------------------
time:   00:21:33
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044817/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044817/

    "2022-09-27T13:22:25.796138+0000 mon.a (mon.0) 1902 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7044826]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:17:23
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044826/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044826/
sentry: https://sentry.ceph.com/organizations/ceph/?query=18b94d936f8445a3b09cbf59d5e49020

    reached maximum tries (90) after waiting for 540 seconds

[7044830]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
-----------------------------------------------------------------
time:   01:00:31
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044830/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044830/

    error during scrub thrashing: reached maximum tries (30) after waiting for
    900 seconds

[7044836]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:05:59
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044836/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044836/
sentry: https://sentry.ceph.com/organizations/ceph/?query=07d746c3f04f43ac956b4593df6b0d11

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1
    =7e76448beefff9399729adb1dc5583efa7528947

[7044843]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:20:10
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044843/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044843/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1b675a66c1ac4659b1eec4ae2e40aeb7

    reached maximum tries (90) after waiting for 540 seconds

[7044850]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   01:51:32
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044850/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044850/

    "2022-09-27T14:13:35.735336+0000 mds.c (mds.0) 6 : cluster [WRN]
    client.4657 isn't responding to mclientcaps(revoke), ino 0x1000000f589
    pending pAsLsXsFsc issued pAsLsXsFscb, sent 304.859199 seconds ago" in
    cluster log

[7044852]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}}
-----------------------------------------------------------------
time:   00:11:25
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044852/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044852/
sentry: https://sentry.ceph.com/organizations/ceph/?query=37492fe2c23146c583133ec328f0ad4e

    Command failed on smithi097 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:7e76448beefff9399729adb1dc5583efa7528947 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4e7fca0c-3e6a-11ed-8431-001a4aab830c -- ceph orch daemon add osd
    smithi097:vg_nvme/lv_3'



Dead
=================================================================
[7044847]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044847/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044847/

    hit max job timeout



Running
=================================================================
[7044839] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044839/



Pass
=================================================================
[7044571] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:15:17
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044571/

[7044572] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:32:09
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044572/

[7044573] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush}
time:   00:10:06
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044573/

[7044574] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:29:20
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044574/

[7044575] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}}
time:   02:01:58
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044575/

[7044576] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:20:42
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044576/

[7044577] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:11:06
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044577/

[7044578] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}}
time:   00:38:10
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044578/

[7044579] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}
time:   00:23:47
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044579/

[7044580] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:14:48
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044580/

[7044581] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}}
time:   01:52:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044581/

[7044582] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:12:47
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044582/

[7044583] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:25:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044583/

[7044584] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry}
time:   00:09:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044584/

[7044586] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}}
time:   00:40:11
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044586/

[7044587] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:42:31
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044587/

[7044588] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
time:   00:23:21
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044588/

[7044590] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:17:15
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044590/

[7044591] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:11:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044591/

[7044592] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i24137}
time:   00:09:30
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044592/

[7044593] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:44:02
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044593/

[7044594] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full}
time:   00:17:17
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044594/

[7044595] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client}
time:   00:09:05
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044595/

[7044596] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror}
time:   01:56:05
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044596/

[7044597] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit}
time:   00:19:42
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044597/

[7044598] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:47:57
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044598/

[7044599] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:56:33
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044599/

[7044600] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:12:43
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044600/

[7044601] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
time:   00:34:42
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044601/

[7044602] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:44:13
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044602/

[7044603] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:10:33
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044603/

[7044604] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{rhel_8} tasks/fstop}
time:   00:08:42
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044604/

[7044605] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:23:17
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044605/

[7044607] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/lockdep}
time:   00:16:15
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044607/

[7044608] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}}
time:   00:17:30
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044608/

[7044609] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}}
time:   00:27:57
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044609/

[7044610] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc}
time:   00:18:27
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044610/

[7044611] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:12:34
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044611/

[7044612] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:35:03
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044612/

[7044613] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:13:51
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044613/

[7044614] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable}
time:   00:10:23
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044614/

[7044615] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
time:   00:21:42
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044615/

[7044616] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:12:11
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044616/

[7044617] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:28:00
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044617/

[7044618] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:44:04
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044618/

[7044619] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
time:   00:33:41
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044619/

[7044620] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm}
time:   00:09:59
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044620/

[7044621] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:41:15
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044621/

[7044623] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:23:33
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044623/

[7044624] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota}
time:   00:11:58
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044624/

[7044625] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:12:49
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044625/

[7044626] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}}
time:   00:22:12
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044626/

[7044627] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}}
time:   00:08:29
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044627/

[7044628] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:15:57
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044628/

[7044629] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:27:00
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044629/

[7044630] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:21:27
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044630/

[7044631] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs}
time:   00:09:24
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044631/

[7044632] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}}
time:   00:24:40
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044632/

[7044633] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:12:08
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044633/

[7044634] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:22:01
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044634/

[7044635] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}}
time:   00:23:39
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044635/

[7044636] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub}
time:   00:17:12
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044636/

[7044637] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:16:41
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044637/

[7044638] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:09:29
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044638/

[7044639] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:11:51
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044639/

[7044640] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:10:40
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044640/

[7044641] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
time:   00:43:51
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044641/

[7044642] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:45:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044642/

[7044643] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap}
time:   00:14:18
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044643/

[7044644] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
time:   00:53:19
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044644/

[7044645] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:35:39
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044645/

[7044646] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:12:08
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044646/

[7044647] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
time:   00:32:19
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044647/

[7044648] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule}
time:   00:45:33
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044648/

[7044649] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:28:06
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044649/

[7044650] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:09:35
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044650/

[7044651] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:19:12
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044651/

[7044652] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:46:52
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044652/

[7044653] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:14:13
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044653/

[7044654] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/valgrind}
time:   00:53:42
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044654/

[7044655] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}}
time:   00:26:55
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044655/

[7044656] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}}
time:   01:47:28
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044656/

[7044657] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:12:13
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044657/

[7044658] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap_schedule_snapdir}
time:   00:09:04
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044658/

[7044660] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}}
time:   00:55:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044660/

[7044661] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:27:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044661/

[7044662] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:22:28
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044662/

[7044663] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots}
time:   00:21:55
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044663/

[7044664] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
time:   00:23:58
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044664/

[7044666] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:16:09
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044666/

[7044667] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}}
time:   00:22:42
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044667/

[7044668] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays}
time:   00:26:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044668/

[7044669] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:12:16
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044669/

[7044670] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:33:45
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044670/

[7044671] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:28:46
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044671/

[7044672] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:14:05
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044672/

[7044673] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration}
time:   00:13:20
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044673/

[7044674] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:13:35
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044674/

[7044675] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/ino_release_cb}
time:   00:08:50
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044675/

[7044676] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:03
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044676/

[7044677] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
time:   00:25:07
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044677/

[7044678] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:12:00
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044678/

[7044679] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:44:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044679/

[7044680] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay}
time:   00:12:57
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044680/

[7044681] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
time:   00:30:00
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044681/

[7044683] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:43:34
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044683/

[7044684] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:47:56
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044684/

[7044685] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:23:17
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044685/

[7044686] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}}
time:   00:34:29
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044686/

[7044687] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:21:00
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044687/

[7044688] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/dir-max-entries}
time:   00:10:02
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044688/

[7044689] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:11:15
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044689/

[7044690] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
time:   00:22:58
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044690/

[7044691] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:13:38
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044691/

[7044692] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota}
time:   00:10:46
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044692/

[7044693] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}}
time:   00:20:58
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044693/

[7044694] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:28:42
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044694/

[7044695] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:11:39
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044695/

[7044696] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
time:   00:13:19
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044696/

[7044697] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
time:   00:10:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044697/

[7044698] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:47:51
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044698/

[7044699] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:38:24
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044699/

[7044701] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/lockdep}
time:   00:11:55
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044701/

[7044703] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}}
time:   00:19:50
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044703/

[7044704] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:20:49
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044704/

[7044706] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   00:49:06
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044706/

[7044707] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:50:17
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044707/

[7044708] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:16:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044708/

[7044709] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}}
time:   01:37:30
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044709/

[7044710] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:18:38
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044710/

[7044711] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:11:35
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044711/

[7044712] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:10:52
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044712/

[7044713] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:09:26
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044713/

[7044714] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:10:57
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044714/

[7044715] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
time:   00:38:06
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044715/

[7044716] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:14:19
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044716/

[7044717] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}}
time:   01:08:30
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044717/

[7044718] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
time:   00:23:47
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044718/

[7044719] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:12:10
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044719/

[7044720] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:28:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044720/

[7044721] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:32:48
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044721/

[7044722] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
time:   01:02:23
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044722/

[7044723] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
time:   00:09:01
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044723/

[7044724] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:54:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044724/

[7044725] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   00:24:33
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044725/

[7044726] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:17:31
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044726/

[7044727] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:21:50
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044727/

[7044728] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace}
time:   00:09:12
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044728/

[7044729] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:15:45
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044729/

[7044730] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}}
time:   00:27:49
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044730/

[7044731] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:28:25
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044731/

[7044732] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:50:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044732/

[7044733] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:10:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044733/

[7044734] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:26:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044734/

[7044735] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush}
time:   00:11:51
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044735/

[7044737] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
time:   00:20:43
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044737/

[7044738] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:13:26
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044738/

[7044739] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:27:45
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044739/

[7044740] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits}
time:   00:17:18
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044740/

[7044741] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   00:53:20
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044741/

[7044742] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   00:33:30
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044742/

[7044743] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:11:43
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044743/

[7044744] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/whitelist_health}
time:   00:15:18
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044744/

[7044745] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}}
time:   00:11:17
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044745/

[7044747] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:25:08
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044747/

[7044748] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:11:02
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044748/

[7044750] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:50:25
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044750/

[7044751] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/valgrind}
time:   00:53:51
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044751/

[7044753] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead}
time:   00:08:39
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044753/

[7044755] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:16:59
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044755/

[7044756] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:24:43
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044756/

[7044757] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:10:26
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044757/

[7044758] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
time:   00:20:27
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044758/

[7044759] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery}
time:   00:32:09
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044759/

[7044760] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:20:32
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044760/

[7044761] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
time:   00:31:15
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044761/

[7044762] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:12:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044762/

[7044763] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage}
time:   00:16:40
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044763/

[7044764] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:13:45
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044764/

[7044765] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}}
time:   00:24:36
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044765/

[7044766] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}}
time:   00:08:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044766/

[7044767] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:25:07
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044767/

[7044768] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
time:   00:51:08
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044768/

[7044769] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan}
time:   00:24:06
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044769/

[7044770] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:48:49
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044770/

[7044771] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
time:   00:40:04
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044771/

[7044772] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:20:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044772/

[7044773] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports}
time:   00:45:45
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044773/

[7044774] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:10:22
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044774/

[7044775] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:34:33
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044775/

[7044776] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
time:   00:31:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044776/

[7044778] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub}
time:   00:12:27
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044778/

[7044779] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}}
time:   01:32:36
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044779/

[7044780] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:11:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044780/

[7044781] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:14:45
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044781/

[7044782] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:12:48
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044782/

[7044784] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
time:   01:02:21
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044784/

[7044786] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:12:48
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044786/

[7044787] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment}
time:   00:14:22
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044787/

[7044788] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:16:07
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044788/

[7044789] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:45:30
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044789/

[7044790] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
time:   00:22:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044790/

[7044791] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:17:44
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044791/

[7044792] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair}
time:   00:18:30
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044792/

[7044793] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:08:48
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044793/

[7044794] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:57:00
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044794/

[7044795] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:31:31
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044795/

[7044796] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:13:29
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044796/

[7044797] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/lockdep}
time:   00:11:33
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044797/

[7044798] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}}
time:   00:16:04
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044798/

[7044799] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}}
time:   00:22:39
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044799/

[7044800] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:10:38
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044800/

[7044801] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:29:02
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044801/

[7044802] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush}
time:   00:09:05
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044802/

[7044803] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:31:29
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044803/

[7044804] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:16:59
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044804/

[7044805] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}}
time:   00:25:24
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044805/

[7044806] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:11:56
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044806/

[7044807] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}
time:   00:19:12
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044807/

[7044808] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   00:56:55
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044808/

[7044809] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   00:29:37
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044809/

[7044810] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:19:49
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044810/

[7044811] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:57:29
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044811/

[7044812] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry}
time:   00:12:46
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044812/

[7044813] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}}
time:   00:40:21
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044813/

[7044814] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:20:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044814/

[7044815] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:10:45
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044815/

[7044816] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
time:   00:21:56
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044816/

[7044818] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:10:51
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044818/

[7044819] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python}
time:   00:09:35
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044819/

[7044820] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:10:39
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044820/

[7044821] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:18:20
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044821/

[7044822] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:27:48
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044822/

[7044823] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
time:   00:21:21
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044823/

[7044824] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:12:24
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044824/

[7044825] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc}
time:   00:15:17
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044825/

[7044827] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}}
time:   00:25:14
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044827/

[7044828] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:34:43
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044828/

[7044829] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable}
time:   00:12:56
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044829/

[7044831] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:21:33
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044831/

[7044832] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:24:11
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044832/

[7044833] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
time:   00:44:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044833/

[7044834] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:23:06
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044834/

[7044835] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm}
time:   00:09:56
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044835/

[7044837] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:19:21
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044837/

[7044838] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:10:26
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044838/

[7044840] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
time:   00:20:20
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044840/

[7044841] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:24:57
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044841/

[7044842] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:13:27
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044842/

[7044844] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/valgrind}
time:   00:55:49
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044844/

[7044845] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}}
time:   00:32:11
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044845/

[7044846] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota}
time:   00:10:01
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044846/

[7044848] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}}
time:   01:30:02
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044848/

[7044849] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:11:40
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044849/

[7044851] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs}
time:   00:11:18
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044851/

[7044853] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:41:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044853/

[7044854] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
time:   00:26:36
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044854/

[7044855] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:13:41
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044855/

[7044856] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:11:53
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044856/

[7044857] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub}
time:   00:17:22
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044857/

[7044858] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:18:52
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044858/

[7044859] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:17:28
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044859/

[7044860] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:12:46
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044860/

[7044861] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}}
time:   00:25:54
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044861/

[7044862] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap}
time:   00:11:46
info:   http://pulpito.ceph.com/teuthology-2022-09-27_03:15:04-fs-quincy-distro-default-smithi/7044862/