2021-11-13T20:15:02.538 INFO:root:teuthology version: 1.1.0-d4737010
2021-11-13T20:15:02.540 INFO:teuthology.results:subset: '111/120000'
2021-11-13T20:15:02.541 INFO:teuthology.results:seed: '7905'
2021-11-14T00:36:55.017 INFO:root:teuthology version: 1.1.0-d4737010
2021-11-14T00:36:55.053 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-11-14T12:36:55.127 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2021-11-14T12:37:12.609 INFO:scrape:Found 162 jobs
2021-11-14T12:37:20.470 INFO:scrape:Found 8 distinct failure reasons
2021-11-14T12:37:20.471 INFO:scrape:Dead: b'2021-11-14T11:11:14.630 DEBUG:teuthology.exit:Finished running handlers'
2021-11-14T12:37:20.472 INFO:scrape:3 jobs: ['6501299', '6501343', '6501258']
2021-11-14T12:37:20.472 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '3-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'tasks/{0-v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-14T12:37:20.473 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-upgrade-with-workload', '3-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'tasks/{0-v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-14T12:37:20.473 INFO:scrape:
2021-11-14T12:37:20.474 INFO:scrape:Failure: {'Failure object was': {'smithi073.front.sepia.ceph.com': {'ansible_facts': {}, 'msg': 'The following modules failed to execute: setup\\n', 'failed_modules': {'setup': {'failed': True, 'module_stdout': '', 'module_stderr': '', 'msg': 'MODULE FAILURE\\nSee stdout/stderr for the exact error', 'rc': -13}}, '_ansible_verbose_override': True, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'failed')"}
2021-11-14T12:37:20.474 INFO:scrape:1 jobs: ['6501312']
2021-11-14T12:37:20.475 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/fsync-tester}', 'wsync/{no}}']
2021-11-14T12:37:20.475 INFO:scrape:
2021-11-14T12:37:20.476 INFO:scrape:Dead: b"2021-11-14T00:58:07.407 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty"
2021-11-14T12:37:20.476 INFO:scrape:1 jobs: ['6501370']
2021-11-14T12:37:20.477 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-upgrade-with-workload', '3-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'tasks/{0-v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-14T12:37:20.477 INFO:scrape:
2021-11-14T12:37:20.478 INFO:scrape:Failure: "2021-11-14T01:02:25.595190+0000 mon.a (mon.0) 376 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2021-11-14T12:37:20.478 INFO:scrape:5 jobs: ['6501375', '6501252', '6501228', '6501304', '6501342']
2021-11-14T12:37:20.479 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}']
2021-11-14T12:37:20.479 INFO:scrape:suites union: ['clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8.stream}', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'fs/functional/{begin', 'fs/snaps/{begin', 'fs/verify/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{mon-debug', 'overrides/{no_client_pidfile', 'overrides/{whitelist_health', 'overrides}}', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/dbench', 'tasks/metrics}', 'tasks/snap-schedule}', 'tasks/workunit/snaps}', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-14T12:37:20.480 INFO:scrape:
2021-11-14T12:37:20.480 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi013 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bed1599fa788bf76a9a9c97632799d018a249f4e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2021-11-14T12:37:20.481 INFO:scrape:1 jobs: ['6501365']
2021-11-14T12:37:20.481 INFO:scrape:suites: ['2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8.stream}', 'fs/thrash/multifs/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-14T12:37:20.482 INFO:scrape:
2021-11-14T12:37:20.482 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
2021-11-14T12:37:20.483 INFO:scrape:1 jobs: ['6501296']
2021-11-14T12:37:20.483 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8.stream}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/client-readahead}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-14T12:37:20.484 INFO:scrape:
2021-11-14T12:37:20.484 INFO:scrape:Failure: Test failure: test_mount_after_evicted_client (tasks.cephfs.test_client_recovery.TestClientRecovery)
2021-11-14T12:37:20.484 INFO:scrape:1 jobs: ['6501298']
2021-11-14T12:37:20.485 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{no_client_pidfile', 'tasks/client-recovery}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-14T12:37:20.485 INFO:scrape:
2021-11-14T12:37:20.486 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2021-11-14T12:37:20.486 INFO:scrape:1 jobs: ['6501217']
2021-11-14T12:37:20.487 INFO:scrape:suites: ['2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-14T12:37:20.487 INFO:scrape:
2021-11-14T12:37:20.495 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/
failed:  9
dead:    4
running: 1
waiting: 0
queued:  0
passed:  148


Fail
=================================================================
[6501217]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:19:25
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501217/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501217/
sentry: https://sentry.ceph.com/organizations/ceph/?query=82ad32e1597f48a4b1aaaeecbbb63e47

    reached maximum tries (90) after waiting for 540 seconds

[6501228]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:19:46
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501228/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501228/

    "2021-11-13T21:59:45.561548+0000 mon.a (mon.0) 322 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[6501252]  fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:51:47
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501252/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501252/

    "2021-11-13T22:37:53.083592+0000 mon.a (mon.0) 207 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[6501296]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead}
-----------------------------------------------------------------
time:   00:11:13
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501296/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501296/
sentry: https://sentry.ceph.com/organizations/ceph/?query=50c3926fb3b94189a163daffa7b500d1

    Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)

[6501298]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:21:51
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501298/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501298/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4a1d951afb2499abf52917cae1e85dd

    Test failure: test_mount_after_evicted_client
    (tasks.cephfs.test_client_recovery.TestClientRecovery)

[6501304]  fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:42:51
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501304/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501304/

    "2021-11-13T23:28:45.484257+0000 mon.a (mon.0) 128 : cluster [WRN] Health
    check failed: Degraded data redundancy: 1/4 objects degraded (25.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6501342]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics}
-----------------------------------------------------------------
time:   00:21:17
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501342/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501342/

    "2021-11-14T00:30:59.959690+0000 mon.a (mon.0) 938 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[6501365]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:30:12
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501365/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501365/
sentry: https://sentry.ceph.com/organizations/ceph/?query=121e68b51a5a4ad9b7c1b047254dabbb

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi013
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bed1599fa788bf76a9a9c97632799d018a249f4e
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[6501375]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:19:52
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501375/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501375/

    "2021-11-14T01:02:25.595190+0000 mon.a (mon.0) 376 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log



Dead
=================================================================
[6501258]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501258/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501258/

    hit max job timeout

[6501299]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501299/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501299/

    hit max job timeout

[6501312]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}}
-----------------------------------------------------------------
time:   00:06:22
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501312/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501312/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b0b753f049b247fb9f1b6e7a12db6bbc

    {'Failure object was': {'smithi073.front.sepia.ceph.com': {'ansible_facts':
    {}, 'msg': 'The following modules failed to execute: setup\\n',
    'failed_modules': {'setup': {'failed': True, 'module_stdout': '',
    'module_stderr': '', 'msg': 'MODULE FAILURE\\nSee stdout/stderr for the
    exact error', 'rc': -13}}, '_ansible_verbose_override': True,
    '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call
    last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-
    ansible_master/callback_plugins/failure_log.py", line 44, in log_failure
    log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com
    _git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/pyt
    hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return
    dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker
    /src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/v
    irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in
    dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 27, in represent node =
    self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut
    hology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/si
    te-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a8509
    9043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a8509
    9043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a8509
    9043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a8509
    9043cf081dc05b4069d301b23fb/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 117, in represent_mapping node_key =
    self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_
    teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/lib/python3.
    6/site-packages/yaml/representer.py", line 58, in represent_data node =
    self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce
    ph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/l
    ib/python3.6/site-packages/yaml/representer.py", line 231, in
    represent_undefined raise RepresenterError("cannot represent an object",
    data)', 'yaml.representer.RepresenterError': "('cannot represent an
    object', 'failed')"}

[6501343]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501343/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501343/

    hit max job timeout



Running
=================================================================
[6501370] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501370/



Pass
=================================================================
[6501214] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
time:   00:13:00
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501214/

[6501215] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}}
time:   00:22:02
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501215/

[6501216] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota}
time:   00:14:08
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501216/

[6501218] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:15:24
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501218/

[6501219] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:07
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501219/

[6501220] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs}
time:   00:12:11
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501220/

[6501221] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   01:53:55
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501221/

[6501222] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}}
time:   00:39:24
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501222/

[6501223] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:22:18
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501223/

[6501224] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub}
time:   00:20:54
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501224/

[6501225] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:14:38
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501225/

[6501226] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap}
time:   00:17:54
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501226/

[6501227] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:18:46
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501227/

[6501229] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}}
time:   00:37:28
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501229/

[6501230] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:38:34
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501230/

[6501231] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots}
time:   00:22:53
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501231/

[6501232] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:14:05
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501232/

[6501233] fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i24137}
time:   00:11:27
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501233/

[6501234] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest}
time:   00:45:59
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501234/

[6501235] fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full}
time:   00:17:14
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501235/

[6501236] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8.stream} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client}
time:   00:11:58
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501236/

[6501237] fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8.stream} tasks/mirror}
time:   01:55:43
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501237/

[6501238] fs/mirror-ha/{begin cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit}
time:   00:20:30
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501238/

[6501239] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:32:42
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501239/

[6501240] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:19:56
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501240/

[6501241] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:32:01
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501241/

[6501242] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:14:15
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501242/

[6501243] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
time:   00:32:54
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501243/

[6501244] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:41:21
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501244/

[6501245] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{centos_8} tasks/fstop}
time:   00:09:59
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501245/

[6501246] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:23:37
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501246/

[6501247] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:25:26
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501247/

[6501248] fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:04:10
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501248/

[6501249] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   00:48:48
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501249/

[6501250] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays}
time:   00:25:47
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501250/

[6501251] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   01:52:54
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501251/

[6501253] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}}
time:   00:27:49
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501253/

[6501254] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration}
time:   00:16:00
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501254/

[6501255] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:18:50
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501255/

[6501256] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay}
time:   00:14:56
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501256/

[6501257] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:19:29
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501257/

[6501259] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota}
time:   00:14:05
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501259/

[6501260] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}}
time:   01:27:10
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501260/

[6501261] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:12:09
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501261/

[6501262] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
time:   00:15:55
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501262/

[6501263] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:13:52
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501263/

[6501264] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:13:26
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501264/

[6501265] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:17:47
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501265/

[6501266] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   00:54:59
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501266/

[6501267] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:35:14
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501267/

[6501268] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:10:39
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501268/

[6501269] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:20:14
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501269/

[6501270] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
time:   01:24:08
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501270/

[6501271] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:14:01
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501271/

[6501272] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}}
time:   00:10:06
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501272/

[6501273] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:56:18
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501273/

[6501274] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:00:01
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501274/

[6501275] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   01:05:59
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501275/

[6501276] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}}
time:   00:39:06
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501276/

[6501277] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
time:   00:22:34
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501277/

[6501278] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:13:12
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501278/

[6501279] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/lockdep}
time:   00:16:33
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501279/

[6501280] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
time:   00:12:17
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501280/

[6501281] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:47
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501281/

[6501282] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   00:15:21
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501282/

[6501283] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:15:30
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501283/

[6501284] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/ino_release_cb}
time:   00:11:12
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501284/

[6501285] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:16:07
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501285/

[6501286] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace}
time:   00:10:02
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501286/

[6501287] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:52:37
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501287/

[6501288] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:19:10
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501288/

[6501289] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush}
time:   00:14:06
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501289/

[6501290] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:10:42
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501290/

[6501291] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}}
time:   00:14:38
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501291/

[6501292] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits}
time:   00:19:49
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501292/

[6501293] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:17:05
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501293/

[6501294] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:30:09
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501294/

[6501295] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
time:   00:19:13
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501295/

[6501297] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:59:30
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501297/

[6501300] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:26:27
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501300/

[6501301] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:08
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501301/

[6501302] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:24:06
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501302/

[6501303] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:11:51
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501303/

[6501305] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:48:03
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501305/

[6501306] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}}
time:   00:17:37
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501306/

[6501307] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage}
time:   00:21:52
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501307/

[6501308] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:09:07
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501308/

[6501309] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan}
time:   00:28:22
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501309/

[6501310] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/valgrind}
time:   00:50:08
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501310/

[6501311] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:21:20
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501311/

[6501313] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports}
time:   00:42:49
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501313/

[6501314] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:20:57
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501314/

[6501315] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:17:07
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501315/

[6501316] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub}
time:   00:17:56
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501316/

[6501317] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:14:21
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501317/

[6501318] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment}
time:   00:19:53
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501318/

[6501319] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:13:17
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501319/

[6501320] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}}
time:   00:14:10
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501320/

[6501321] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:33:58
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501321/

[6501322] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:20:48
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501322/

[6501323] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:49
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501323/

[6501324] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
time:   00:22:25
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501324/

[6501325] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:15:33
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501325/

[6501326] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair}
time:   00:18:39
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501326/

[6501327] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:24:56
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501327/

[6501328] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush}
time:   00:11:00
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501328/

[6501329] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:21:53
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501329/

[6501330] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:48:52
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501330/

[6501331] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:10:04
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501331/

[6501332] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:17:13
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501332/

[6501333] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}}
time:   00:21:24
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501333/

[6501334] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}}
time:   00:26:40
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501334/

[6501335] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}
time:   00:22:15
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501335/

[6501336] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:16:04
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501336/

[6501337] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry}
time:   00:16:42
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501337/

[6501338] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:16:37
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501338/

[6501339] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind}
time:   01:30:39
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501339/

[6501340] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:20
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501340/

[6501341] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:14:48
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501341/

[6501344] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}}
time:   00:21:01
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501344/

[6501345] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   01:43:19
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501345/

[6501346] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc}
time:   00:17:22
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501346/

[6501347] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:35:01
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501347/

[6501348] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:10:16
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501348/

[6501349] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:42:37
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501349/

[6501350] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable}
time:   00:14:27
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501350/

[6501351] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:16:46
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501351/

[6501352] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}}
time:   00:15:13
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501352/

[6501353] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm}
time:   00:15:08
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501353/

[6501354] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:19:08
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501354/

[6501355] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota}
time:   00:13:28
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501355/

[6501356] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   01:37:32
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501356/

[6501357] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:11:47
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501357/

[6501358] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python}
time:   00:14:29
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501358/

[6501359] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:21:32
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501359/

[6501360] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:11:39
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501360/

[6501361] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:38:15
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501361/

[6501362] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:22:49
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501362/

[6501363] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   00:47:12
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501363/

[6501364] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs}
time:   00:11:07
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501364/

[6501366] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}}
time:   00:13:09
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501366/

[6501367] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub}
time:   00:18:47
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501367/

[6501368] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:20:18
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501368/

[6501369] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap}
time:   00:16:35
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501369/

[6501371] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/lockdep}
time:   01:28:10
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501371/

[6501372] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:20:18
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501372/

[6501373] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:48:51
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501373/

[6501374] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
time:   00:12:34
info:   http://pulpito.ceph.com/yuriw-2021-11-13_14:31:12-fs-wip-yuriw-master-11.12.21-distro-basic-smithi/6501374/