2022-12-03T16:46:38.164 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf
2022-12-03T16:46:38.165 INFO:teuthology.results:subset: '111/120000'
2022-12-03T16:46:38.165 INFO:teuthology.results:seed: '2522'
2022-12-03T19:37:30.123 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf
2022-12-03T19:37:30.153 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-12-04T03:03:10.151 INFO:teuthology.results:Tests finished! gathering results...
2022-12-04T03:03:26.108 INFO:scrape:Found 169 jobs
2022-12-04T03:03:28.125 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102142/teuthology.log
2022-12-04T03:03:37.737 INFO:scrape:Found 13 distinct failure reasons
2022-12-04T03:03:37.739 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi080 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2022-12-04T03:03:37.739 INFO:scrape:10 jobs: ['7102071', '7102091', '7102177', '7102117', '7102048', '7102042', '7102173', '7102111', '7102030', '7102158']
2022-12-04T03:03:37.740 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.740 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/snaps/{begin', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'overrides/{whitelist_health', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.741 INFO:scrape:
2022-12-04T03:03:37.741 INFO:scrape:Failure: "2022-12-03T17:38:40.578571+0000 mon.a (mon.0) 3111 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022-12-04T03:03:37.742 INFO:scrape:6 jobs: ['7102032', '7102145', '7102184', '7102176', '7102093', '7102060']
2022-12-04T03:03:37.742 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{no_client_pidfile', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.743 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/asok_dump_tree}', 'tasks/damage}', 'tasks/exports}', 'tasks/metrics}', 'tasks/snap-schedule}', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.743 INFO:scrape:
2022-12-04T03:03:37.744 INFO:scrape:Failure: Stale jobs detected, aborting.
2022-12-04T03:03:37.744 INFO:scrape:2 jobs: ['7102118', '7102114']
2022-12-04T03:03:37.745 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.746 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/no}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{multimds/no', 'overrides/{no_client_pidfile', 'pg-warn', 'rhel_8}', 'tasks/test_journal_migration}', 'tasks/{0-nautilus', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.746 INFO:scrape:
2022-12-04T03:03:37.746 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2022-12-04T03:03:37.747 INFO:scrape:1 jobs: ['7102180']
2022-12-04T03:03:37.747 INFO:scrape:suites: ['2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.748 INFO:scrape:
2022-12-04T03:03:37.748 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2022-12-04T03:03:37.749 INFO:scrape:1 jobs: ['7102129']
2022-12-04T03:03:37.749 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/acls}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.750 INFO:scrape:
2022-12-04T03:03:37.750 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi002 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'
2022-12-04T03:03:37.751 INFO:scrape:1 jobs: ['7102169']
2022-12-04T03:03:37.751 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/1', 'rhel_8}', 'scrub/no', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/fsx}', 'wsync/{no}}']
2022-12-04T03:03:37.752 INFO:scrape:
2022-12-04T03:03:37.752 INFO:scrape:Dead: None
2022-12-04T03:03:37.752 INFO:scrape:2 jobs: ['7102142', '7102116']
2022-12-04T03:03:37.753 INFO:scrape:
2022-12-04T03:03:37.753 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)
2022-12-04T03:03:37.754 INFO:scrape:1 jobs: ['7102049']
2022-12-04T03:03:37.754 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/mds-flush}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.755 INFO:scrape:
2022-12-04T03:03:37.755 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi036.front.sepia.ceph.com: ['type=AVC msg=audit(1670096870.115:16943): avc:  denied  { read } for  pid=703 comm="sssd_be" name="resolv.conf" dev="sda1" ino=524374 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1670096870.115:16943): avc:  denied  { open } for  pid=703 comm="sssd_be" path="/etc/resolv.conf" dev="sda1" ino=524374 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1']
2022-12-04T03:03:37.756 INFO:scrape:2 jobs: ['7102183', '7102171']
2022-12-04T03:03:37.756 INFO:scrape:suites intersection: ['conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.757 INFO:scrape:suites union: ['clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/multifs/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'overrides/{mon-debug', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/multifs-auth}', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/fsync-tester}', 'wsync/{yes}}']
2022-12-04T03:03:37.757 INFO:scrape:
2022-12-04T03:03:37.757 INFO:scrape:Failure: {'smithi008.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi008.front.sepia.ceph.com port 22: No route to host'}}
2022-12-04T03:03:37.758 INFO:scrape:2 jobs: ['7102119', '7102132']
2022-12-04T03:03:37.758 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.759 INFO:scrape:suites union: ['1-upgrade}}', 'centos_latest', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/upgrade/nofs/{bluestore-bitmap', 'fs/verify/{begin', 'mds', 'mon', 'mount/fuse', 'no-mds-cluster', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{mon-debug', 'overrides/{pg-warn', 'ranks/1', 'session_timeout', 'tasks/dbench', 'tasks/{0-octopus', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.759 INFO:scrape:
2022-12-04T03:03:37.760 INFO:scrape:Failure: {'smithi138.front.sepia.ceph.com': {'changed': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi138.front.sepia.ceph.com port 22: Connection refused', 'unreachable': True}}
2022-12-04T03:03:37.760 INFO:scrape:1 jobs: ['7102115']
2022-12-04T03:03:37.761 INFO:scrape:suites: ['centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2022-12-04T03:03:37.761 INFO:scrape:
2022-12-04T03:03:37.762 INFO:scrape:Failure: Cannot connect to remote host smithi138
2022-12-04T03:03:37.763 INFO:scrape:1 jobs: ['7102128']
2022-12-04T03:03:37.764 INFO:scrape:suites: ['2-workunit/fs/trivial_sync}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.765 INFO:scrape:
2022-12-04T03:03:37.766 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_orphan_clone (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2022-12-04T03:03:37.766 INFO:scrape:2 jobs: ['7102159', '7102031']
2022-12-04T03:03:37.767 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.768 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-12-04T03:03:37.769 INFO:scrape:
2022-12-04T03:03:37.781 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/
failed:  29
dead:    3
running: 0
waiting: 0
queued:  0
passed:  137


Fail
=================================================================
[7102030]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:27:22
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102030/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102030/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d00dd100b7df4c5a8e584734c10054e7

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi002
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102031]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:49:22
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102031/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102031/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d42b20163da44c81a1b0177c56266320

    Test failure: test_subvolume_snapshot_info_if_orphan_clone
    (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)

[7102032]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports}
-----------------------------------------------------------------
time:   00:52:21
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102032/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102032/

    "2022-12-03T17:38:40.578571+0000 mon.a (mon.0) 3111 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7102042]  fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:16:40
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102042/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102042/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f4a393abc0284c6396f58c05dcd23974

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi017
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102048]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:19:46
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102048/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102048/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eb282c6a3bc745438714323c739686a5

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi163
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102049]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush}
-----------------------------------------------------------------
time:   00:14:59
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102049/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102049/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f254649dfd1a4291b68bc6ec84b09eb9

    Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)

[7102060]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics}
-----------------------------------------------------------------
time:   00:38:40
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102060/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102060/

    "2022-12-03T18:17:39.537107+0000 mon.a (mon.0) 860 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7102071]  fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:15:18
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102071/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102071/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c5cde725835f43d8b5a43a55f5534689

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi080
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102091]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:18:16
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102091/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102091/
sentry: https://sentry.ceph.com/organizations/ceph/?query=32e89ea6e2614a218e79079602c4a840

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi093
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102093]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:55:49
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102093/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102093/

    "2022-12-03T19:19:11.054383+0000 mon.a (mon.0) 1374 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7102111]  fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:14:05
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102111/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102111/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1dacb6bd9996402bb2f70bc01d339d8f

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi136
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102114]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102114/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102114/
sentry: https://sentry.ceph.com/organizations/ceph/?query=09ffe98e9c2e482d9f06989fbf4c2dd2

    Stale jobs detected, aborting.

[7102115]  fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   00:07:30
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102115/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102115/
sentry: https://sentry.ceph.com/organizations/ceph/?query=06c755cbed5c414798d1ca410623c7d7

    {'smithi138.front.sepia.ceph.com': {'changed': False, 'msg': 'Failed to
    connect to the host via ssh: ssh: connect to host
    smithi138.front.sepia.ceph.com port 22: Connection refused', 'unreachable':
    True}}

[7102117]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:19:20
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102117/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102117/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0a33aa7bdd2b4a29bae0457dec68b8d3

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi072
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102118]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102118/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102118/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e62758338ba4fb08cc64531a68875ac

    Stale jobs detected, aborting.

[7102128]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102128/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102128/
sentry: https://sentry.ceph.com/organizations/ceph/?query=12d65954bd45431f897f850290ae0611

    Cannot connect to remote host smithi138

[7102129]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
-----------------------------------------------------------------
time:   00:12:34
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102129/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102129/
sentry: https://sentry.ceph.com/organizations/ceph/?query=53316072f9cc48ae8a283dbf5c7d7f59

    Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)

[7102132]  fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}}
-----------------------------------------------------------------
time:   00:04:51
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102132/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102132/
sentry: https://sentry.ceph.com/organizations/ceph/?query=467d121c975d4b6681d1dea164795683

    {'smithi008.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg':
    'Failed to connect to the host via ssh: ssh: connect to host
    smithi008.front.sepia.ceph.com port 22: No route to host'}}

[7102145]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
-----------------------------------------------------------------
time:   00:20:04
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102145/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102145/

    "2022-12-03T19:09:41.780591+0000 mon.a (mon.0) 450 : cluster [WRN] Health
    check failed: Degraded data redundancy: 28/1894 objects degraded (1.478%),
    1 pg degraded (PG_DEGRADED)" in cluster log

[7102158]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:25:33
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102158/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102158/
sentry: https://sentry.ceph.com/organizations/ceph/?query=91cca2a2cff34248a7f278930e2d08f6

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi008
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102159]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   01:08:54
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102159/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102159/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d9b90989aa77420195e0b7e5e44e147d

    Test failure: test_subvolume_snapshot_info_if_orphan_clone
    (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)

[7102169]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:37
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102169/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102169/
sentry: https://sentry.ceph.com/organizations/ceph/?query=47c8965ebb004619aa849c7768641ac7

    Command failed (workunit test suites/fsx.sh) on smithi002 with status 128:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'

[7102171]  fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:32:45
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102171/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102171/

    SELinux denials found on ubuntu@smithi130.front.sepia.ceph.com: ['type=AVC
    msg=audit(1670096141.344:16943): avc:  denied  { read } for  pid=804
    comm="sssd_be" name="resolv.conf" dev="sda1" ino=524374
    scontext=system_u:system_r:sssd_t:s0
    tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1',
    'type=AVC msg=audit(1670096141.344:16943): avc:  denied  { open } for
    pid=804 comm="sssd_be" path="/etc/resolv.conf" dev="sda1" ino=524374
    scontext=system_u:system_r:sssd_t:s0
    tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1']

[7102173]  fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:21:11
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102173/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102173/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ecef459864324c7e9f6754a32e092f9e

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi090
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102176]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage}
-----------------------------------------------------------------
time:   00:25:52
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102176/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102176/

    "2022-12-03T19:42:22.095769+0000 mon.a (mon.0) 214 : cluster [WRN] Health
    check failed: Degraded data redundancy: 3/44 objects degraded (6.818%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[7102177]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:25:09
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102177/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102177/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3f307aa026f64d10a4977214729be60d

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi115
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7102180]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   07:32:10
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102180/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102180/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d918f1280fb4cba8d4a82502497e885

    Command failed (workunit test suites/fsstress.sh) on smithi061 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5a8f55c8d279c87c9c1137aaa5063e74eb17fa09
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7102183]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
-----------------------------------------------------------------
time:   00:20:56
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102183/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102183/

    SELinux denials found on ubuntu@smithi036.front.sepia.ceph.com: ['type=AVC
    msg=audit(1670096870.115:16943): avc:  denied  { read } for  pid=703
    comm="sssd_be" name="resolv.conf" dev="sda1" ino=524374
    scontext=system_u:system_r:sssd_t:s0
    tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1',
    'type=AVC msg=audit(1670096870.115:16943): avc:  denied  { open } for
    pid=703 comm="sssd_be" path="/etc/resolv.conf" dev="sda1" ino=524374
    scontext=system_u:system_r:sssd_t:s0
    tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1']

[7102184]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:24:30
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102184/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102184/

    "2022-12-03T19:51:52.268890+0000 mon.a (mon.0) 455 : cluster [WRN] Health
    check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY)"
    in cluster log



Dead
=================================================================
[7102116]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volume-client}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102116/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102116/

    Error reimaging machines: Failed to power on smithi120

[7102119]  fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:09:41
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102119/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102119/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d28b50aa2d9407f9989bb624b8a5433

    {'smithi008.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg':
    'Failed to connect to the host via ssh: ssh: connect to host
    smithi008.front.sepia.ceph.com port 22: No route to host'}}

[7102142]  fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102142/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102142/

    Error reimaging machines: reached maximum tries (60) after waiting for 900
    seconds



Pass
=================================================================
[7102028] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:24:13
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102028/

[7102029] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan}
time:   00:34:12
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102029/

[7102033] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   00:28:59
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102033/

[7102034] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   00:23:10
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102034/

[7102035] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:23:17
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102035/

[7102036] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub}
time:   00:15:55
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102036/

[7102037] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment}
time:   00:18:11
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102037/

[7102038] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:39:43
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102038/

[7102039] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:18:48
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102039/

[7102040] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:52:35
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102040/

[7102041] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:15:33
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102041/

[7102043] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:58:21
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102043/

[7102044] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:30:23
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102044/

[7102045] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/valgrind}
time:   00:51:42
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102045/

[7102046] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair}
time:   00:25:31
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102046/

[7102047] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}}
time:   00:37:47
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102047/

[7102050] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:38:09
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102050/

[7102051] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:11:35
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102051/

[7102052] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:22:33
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102052/

[7102053] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}
time:   00:22:09
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102053/

[7102054] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:33:17
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102054/

[7102055] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}}
time:   00:28:04
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102055/

[7102056] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:26:00
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102056/

[7102057] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
time:   00:17:46
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102057/

[7102058] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry}
time:   00:14:58
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102058/

[7102059] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:15:40
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102059/

[7102061] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:13:21
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102061/

[7102062] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python}
time:   00:12:43
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102062/

[7102063] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:13:33
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102063/

[7102064] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:20:32
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102064/

[7102065] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc}
time:   00:22:03
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102065/

[7102066] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:17:09
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102066/

[7102067] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
time:   00:14:14
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102067/

[7102068] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable}
time:   00:13:11
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102068/

[7102069] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:29:49
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102069/

[7102070] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:31:01
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102070/

[7102072] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:16:45
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102072/

[7102073] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:30:30
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102073/

[7102074] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm}
time:   00:12:40
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102074/

[7102075] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/lockdep}
time:   00:18:36
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102075/

[7102076] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:23:15
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102076/

[7102077] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:38:12
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102077/

[7102078] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
time:   00:13:45
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102078/

[7102079] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota}
time:   00:19:50
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102079/

[7102080] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}}
time:   00:14:07
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102080/

[7102081] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}}
time:   00:39:00
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102081/

[7102082] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:20:42
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102082/

[7102083] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs}
time:   00:15:10
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102083/

[7102084] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:43
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102084/

[7102085] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:48
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102085/

[7102086] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:26:57
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102086/

[7102087] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub}
time:   00:19:27
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102087/

[7102088] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:28:57
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102088/

[7102089] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
time:   00:47:02
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102089/

[7102090] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap}
time:   00:15:02
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102090/

[7102092] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:54
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102092/

[7102094] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:16:48
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102094/

[7102095] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots}
time:   00:26:53
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102095/

[7102096] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
time:   00:39:43
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102096/

[7102097] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:23:14
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102097/

[7102098] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays}
time:   00:30:20
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102098/

[7102099] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:27
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102099/

[7102100] fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i24137}
time:   00:13:49
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102100/

[7102101] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest}
time:   00:47:23
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102101/

[7102102] fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full}
time:   00:21:44
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102102/

[7102103] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client}
time:   00:13:17
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102103/

[7102104] fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror}
time:   01:58:08
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102104/

[7102105] fs/mirror-ha/{begin cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit}
time:   02:37:33
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102105/

[7102106] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:31:35
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102106/

[7102107] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:48:13
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102107/

[7102108] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:30:08
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102108/

[7102109] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:15:29
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102109/

[7102110] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
time:   00:36:47
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102110/

[7102112] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop}
time:   00:09:40
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102112/

[7102113] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:22:42
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102113/

[7102120] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}}
time:   00:21:31
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102120/

[7102121] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay}
time:   00:19:25
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102121/

[7102122] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:42:57
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102122/

[7102123] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:59
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102123/

[7102124] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/dir-max-entries}
time:   00:17:51
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102124/

[7102125] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:19:02
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102125/

[7102126] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota}
time:   00:14:12
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102126/

[7102127] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}}
time:   01:14:43
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102127/

[7102130] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:34
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102130/

[7102131] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:01
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102131/

[7102133] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:25:13
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102133/

[7102134] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   01:01:35
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102134/

[7102135] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   01:32:49
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102135/

[7102136] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:40:20
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102136/

[7102137] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:11:50
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102137/

[7102138] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:17:42
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102138/

[7102139] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
time:   00:43:40
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102139/

[7102140] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:15:42
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102140/

[7102141] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:52:10
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102141/

[7102143] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:56:52
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102143/

[7102144] fs/upgrade/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04}
time:   01:13:44
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102144/

[7102146] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:17:09
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102146/

[7102147] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/lockdep}
time:   00:19:23
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102147/

[7102148] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
time:   00:13:45
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102148/

[7102149] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:56
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102149/

[7102150] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
time:   00:19:02
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102150/

[7102151] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:14
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102151/

[7102152] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/ino_release_cb}
time:   00:11:08
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102152/

[7102153] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:15:26
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102153/

[7102154] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:30:26
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102154/

[7102155] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace}
time:   00:18:35
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102155/

[7102156] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:33:03
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102156/

[7102157] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush}
time:   00:15:06
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102157/

[7102160] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits}
time:   00:21:38
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102160/

[7102161] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}}
time:   00:16:24
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102161/

[7102162] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:21:12
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102162/

[7102163] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead}
time:   00:12:06
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102163/

[7102164] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:16:58
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102164/

[7102165] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:35:10
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102165/

[7102166] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
time:   00:18:23
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102166/

[7102167] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery}
time:   00:39:45
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102167/

[7102168] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:42:18
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102168/

[7102170] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:17
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102170/

[7102172] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:07
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102172/

[7102174] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:49:02
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102174/

[7102175] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:32:45
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102175/

[7102178] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan}
time:   00:26:13
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102178/

[7102179] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/valgrind}
time:   00:54:14
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102179/

[7102181] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports}
time:   00:45:45
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102181/

[7102182] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:25:06
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102182/

[7102185] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:21:57
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102185/

[7102186] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub}
time:   00:23:27
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102186/

[7102187] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment}
time:   00:22:24
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102187/

[7102188] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:13:19
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102188/

[7102189] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:39
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102189/

[7102190] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}}
time:   00:15:38
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102190/

[7102191] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:27:41
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102191/

[7102192] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:49:42
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102192/

[7102193] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:51
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102193/

[7102194] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:30:58
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102194/

[7102195] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
time:   00:26:44
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102195/

[7102196] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair}
time:   00:19:23
info:   http://pulpito.ceph.com/yuriw-2022-12-03_16:37:49-fs-wip-yuri2-testing-2022-12-02-1232-pacific-distro-default-smithi/7102196/