2023-04-25T18:06:12.282 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-25T18:06:12.306 INFO:teuthology.results:subset: '111/12000'
2023-04-25T18:06:12.307 INFO:teuthology.results:seed: '2155'
2023-04-26T02:19:28.548 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-26T02:19:28.591 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-26T12:57:24.517 INFO:teuthology.results:Tests finished! gathering results...
2023-04-26T12:57:38.751 INFO:scrape:Found 172 jobs
2023-04-26T12:57:49.801 INFO:scrape:Found 12 distinct failure reasons
2023-04-26T12:57:49.802 INFO:scrape:Failure: more than one file system available
2023-04-26T12:57:49.803 INFO:scrape:15 jobs: ['7251413', '7251357', '7251451', '7251573', '7251619', '7251490', '7251535', '7251311', '7251431', '7251387', '7251475', '7251371', '7251513', '7251295', '7251601']
2023-04-26T12:57:49.803 INFO:scrape:suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health}']
2023-04-26T12:57:49.804 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-04-26T12:57:49.804 INFO:scrape:
2023-04-26T12:57:49.804 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_sync (tasks.cephfs.test_mirroring.TestMirroring)
2023-04-26T12:57:49.804 INFO:scrape:2 jobs: ['7251349', '7251327']
2023-04-26T12:57:49.805 INFO:scrape:suites intersection: ['mount/fuse']
2023-04-26T12:57:49.805 INFO:scrape:suites union: ['centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/whitelist_health', 'overrides/{whitelist_health}', 'supported-random-distros$/{centos_8}', 'tasks/mirror}', 'tasks/mirror}}']
2023-04-26T12:57:49.805 INFO:scrape:
2023-04-26T12:57:49.806 INFO:scrape:Failure: "2023-04-25T22:52:37.659326+0000 mon.a (mon.0) 1031 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
2023-04-26T12:57:49.806 INFO:scrape:3 jobs: ['7251486', '7251389', '7251597']
2023-04-26T12:57:49.806 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health']
2023-04-26T12:57:49.807 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/snap-schedule}', 'tasks/volumes/{overrides', 'test/basic}}', 'test/misc}}']
2023-04-26T12:57:49.807 INFO:scrape:
2023-04-26T12:57:49.807 INFO:scrape:Failure: Command failed on smithi053 with status 1: 'sudo yum upgrade -y linux-firmware'
2023-04-26T12:57:49.808 INFO:scrape:2 jobs: ['7251569', '7251555']
2023-04-26T12:57:49.808 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest']
2023-04-26T12:57:49.808 INFO:scrape:suites union: ['2-workunit/ffsb}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'tasks/scrub}', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-04-26T12:57:49.808 INFO:scrape:
2023-04-26T12:57:49.809 INFO:scrape:Dead: b'2023-04-26T12:52:41.379 DEBUG:teuthology.exit:Finished running handlers'
2023-04-26T12:57:49.809 INFO:scrape:3 jobs: ['7251571', '7251437', '7251303']
2023-04-26T12:57:49.809 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2023-04-26T12:57:49.810 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2023-04-26T12:57:49.810 INFO:scrape:
2023-04-26T12:57:49.810 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-04-26T12:57:49.811 INFO:scrape:3 jobs: ['7251314', '7251579', '7251393']
2023-04-26T12:57:49.811 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-04-26T12:57:49.811 INFO:scrape:suites union: ['2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'fs/verify/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'session_timeout}', 'tasks/fsstress', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'validater/lockdep}']
2023-04-26T12:57:49.812 INFO:scrape:
2023-04-26T12:57:49.812 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi087 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b35a461e1c1b935b2b3fc7c43d68a58c1a41547 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-04-26T12:57:49.812 INFO:scrape:1 jobs: ['7251545']
2023-04-26T12:57:49.812 INFO:scrape:suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-04-26T12:57:49.813 INFO:scrape:
2023-04-26T12:57:49.813 INFO:scrape:Failure: {'smithi135.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/655edd281b923f12af44ba71f23c34741aa8dae0a29e712e75d96be8787f7115-modules.yaml.xz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}}
2023-04-26T12:57:49.813 INFO:scrape:4 jobs: ['7251629', '7251459', '7251477', '7251457']
2023-04-26T12:57:49.814 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-04-26T12:57:49.814 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'fs/permission/{begin', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/no', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/cfuse_workunit_misc}', 'tasks/data-scan}', 'tasks/forward-scrub}', 'tasks/{0-nautilus']
2023-04-26T12:57:49.814 INFO:scrape:
2023-04-26T12:57:49.815 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2023-04-26T12:57:49.815 INFO:scrape:1 jobs: ['7251521']
2023-04-26T12:57:49.815 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}']
2023-04-26T12:57:49.816 INFO:scrape:
2023-04-26T12:57:49.816 INFO:scrape:Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)
2023-04-26T12:57:49.816 INFO:scrape:1 jobs: ['7251511']
2023-04-26T12:57:49.816 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/mds-full}']
2023-04-26T12:57:49.817 INFO:scrape:
2023-04-26T12:57:49.817 INFO:scrape:Failure: Test failure: test_delayed_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2023-04-26T12:57:49.817 INFO:scrape:1 jobs: ['7251523']
2023-04-26T12:57:49.818 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/metrics}']
2023-04-26T12:57:49.818 INFO:scrape:
2023-04-26T12:57:49.818 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2023-04-26T12:57:49.818 INFO:scrape:1 jobs: ['7251367']
2023-04-26T12:57:49.819 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}']
2023-04-26T12:57:49.819 INFO:scrape:
2023-04-26T12:57:49.828 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/
failed:  30
dead:    7
running: 0
waiting: 0
queued:  0
passed:  135


Fail
=================================================================
[7251295]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:16:31
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251295/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251295/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0b4a9160c7ae46d980f582bc1e0fafbc

    more than one file system available

[7251311]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:22:06
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251311/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251311/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a36b9f56964146a3b3cc27cb551a4c9e

    more than one file system available

[7251314]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   07:16:13
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251314/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251314/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d63699acb4ee4bc297d346dbc659deb4

    Command failed (workunit test suites/fsstress.sh) on smithi031 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5b35a461e1c1b935b2b3fc7c43d68a58c1a41547
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7251327]  fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror}
-----------------------------------------------------------------
time:   01:23:18
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251327/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251327/
sentry: https://sentry.ceph.com/organizations/ceph/?query=97ec4294ce4342a594e050b19a765790

    Test failure: test_cephfs_mirror_cancel_sync
    (tasks.cephfs.test_mirroring.TestMirroring)

[7251349]  fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   01:34:30
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251349/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251349/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a787291703f043e7a633e0f5ce3bdb5f

    Test failure: test_cephfs_mirror_cancel_sync
    (tasks.cephfs.test_mirroring.TestMirroring)

[7251357]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:14:25
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251357/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251357/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13644a2bf92a4f76980adf7a17ed97df

    more than one file system available

[7251367]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
-----------------------------------------------------------------
time:   00:13:39
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251367/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251367/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a8eaf62e9a4844f086af51f18b1f2967

    Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)

[7251371]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:19:56
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251371/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251371/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0f3f89e9b8f34650a9b697bb544edf0d

    more than one file system available

[7251387]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   01:09:29
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251387/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251387/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bf5b7ecee9dd42c88925170c37111524

    more than one file system available

[7251389]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   01:32:51
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251389/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251389/

    "2023-04-25T21:15:43.837510+0000 mon.a (mon.0) 5341 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7251393]  fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   06:17:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251393/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251393/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1069c23742c6422cb84f2e6023f8bed7

    Command failed (workunit test suites/fsstress.sh) on smithi134 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5b35a461e1c1b935b2b3fc7c43d68a58c1a41547
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7251413]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:16:18
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251413/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251413/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5760daed46b54afcbfb104d3d770dd7e

    more than one file system available

[7251431]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   01:06:58
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251431/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251431/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ff3e302e80f3450e98069232d01cbeeb

    more than one file system available

[7251451]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:15:58
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251451/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251451/
sentry: https://sentry.ceph.com/organizations/ceph/?query=952cc1bf486545908993b881e2752246

    more than one file system available

[7251475]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   01:13:47
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251475/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251475/
sentry: https://sentry.ceph.com/organizations/ceph/?query=209e656a61fd4d16be1a35ff0edd56f6

    more than one file system available

[7251486]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:22:29
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251486/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251486/

    "2023-04-25T22:52:37.659326+0000 mon.a (mon.0) 1031 : cluster [WRN] Health
    check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)"
    in cluster log

[7251490]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:16:42
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251490/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251490/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e788fe9737b460fa3252d4e36686def

    more than one file system available

[7251511]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
-----------------------------------------------------------------
time:   01:34:56
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251511/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251511/
sentry: https://sentry.ceph.com/organizations/ceph/?query=429e26889f844db3994c768a3dd6fe5e

    Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)

[7251513]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   01:10:59
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251513/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251513/
sentry: https://sentry.ceph.com/organizations/ceph/?query=af1e1ebf52b64d3686db0f62b69e106b

    more than one file system available

[7251521]  fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:48:30
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251521/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251521/

    reached maximum tries (50) after waiting for 300 seconds

[7251523]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:19:12
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251523/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251523/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fdd09fdc538c421e95040f540bd33d7d

    Test failure: test_delayed_metrics
    (tasks.cephfs.test_mds_metrics.TestMDSMetrics)

[7251535]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:24:15
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251535/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251535/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b2fdfb6ad8b54084b169c41126d6d5ad

    more than one file system available

[7251545]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:20:23
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251545/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251545/
sentry: https://sentry.ceph.com/organizations/ceph/?query=67587bd8b7c547b1abcf74a605186843

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi087
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5b35a461e1c1b935b2b3fc7c43d68a58c1a41547
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7251555]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251555/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251555/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3eeb00b9b1154ea3a866a4f7b6838849

    Command failed on smithi165 with status 1: 'sudo yum upgrade -y linux-
    firmware'

[7251569]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251569/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251569/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6dba3b68b8ad4b7e85e7bf36f790017e

    Command failed on smithi053 with status 1: 'sudo yum upgrade -y linux-
    firmware'

[7251573]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:16:12
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251573/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251573/
sentry: https://sentry.ceph.com/organizations/ceph/?query=42d0fb99943a4e4eae96f2f33890f08f

    more than one file system available

[7251579]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   06:46:51
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251579/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251579/
sentry: https://sentry.ceph.com/organizations/ceph/?query=871a1e03e1424f2197cb1bb236e8475c

    Command failed (workunit test suites/fsstress.sh) on smithi100 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=5b35a461e1c1b935b2b3fc7c43d68a58c1a41547
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7251597]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   01:42:34
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251597/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251597/

    "2023-04-26T02:57:16.605232+0000 mon.a (mon.0) 389 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7251601]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:16:23
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251601/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251601/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ff3ce5a732ef4975be6081d429d51528

    more than one file system available

[7251619]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:14:08
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251619/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251619/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d556dddf5f9c4b8fab2f4e9d0bd1acb9

    more than one file system available



Dead
=================================================================
[7251303]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251303/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251303/

    hit max job timeout

[7251437]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251437/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251437/

    hit max job timeout

[7251457]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:33:54
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251457/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251457/
sentry: https://sentry.ceph.com/organizations/ceph/?query=00db22d1aac145afb4ef7b0ffc16ee68

    {'smithi078.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/655edd281b923f12
    af44ba71f23c34741aa8dae0a29e712e75d96be8787f7115-modules.yaml.xz - Cannot
    download, all mirrors were already tried without success", 'rc': 1,
    'results': []}}

[7251459]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
-----------------------------------------------------------------
time:   00:33:50
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251459/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251459/
sentry: https://sentry.ceph.com/organizations/ceph/?query=192836bf8a8c48ca8748e81a53280d2f

    {'smithi042.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24
    680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot
    download, all mirrors were already tried without success; repodata/144bb6d0
    3f4cceafda5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz -
    Cannot download, all mirrors were already tried without success;
    repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-
    comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already
    tried without success; repodata/655edd281b923f12af44ba71f23c34741aa8dae0a29
    e712e75d96be8787f7115-modules.yaml.xz - Cannot download, all mirrors were
    already tried without success", 'rc': 1, 'results': []}}

[7251477]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:25:33
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251477/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251477/
sentry: https://sentry.ceph.com/organizations/ceph/?query=63fa5f4a3ba241f9a7d15838e10d032e

    {'smithi032.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/144bb6d03f4cceaf
    da5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot
    download, all mirrors were already tried without success; repodata/655edd28
    1b923f12af44ba71f23c34741aa8dae0a29e712e75d96be8787f7115-modules.yaml.xz -
    Cannot download, all mirrors were already tried without success", 'rc': 1,
    'results': []}, 'smithi059.front.sepia.ceph.com': {'_ansible_no_log':
    False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade':
    False, 'allowerasing': False, 'autoremove': False, 'bugfix': False,
    'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/655edd281b923f12
    af44ba71f23c34741aa8dae0a29e712e75d96be8787f7115-modules.yaml.xz - Cannot
    download, all mirrors were already tried without success", 'rc': 1,
    'results': []}}

[7251571]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251571/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251571/

    hit max job timeout

[7251629]  fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:33:46
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251629/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251629/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f56a3cb75ca04371a2a0baa61cbda2ed

    {'smithi135.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/655edd281b923f12
    af44ba71f23c34741aa8dae0a29e712e75d96be8787f7115-modules.yaml.xz - Cannot
    download, all mirrors were already tried without success", 'rc': 1,
    'results': []}}



Pass
=================================================================
[7251291] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:55:50
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251291/

[7251293] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
time:   00:15:53
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251293/

[7251297] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   01:06:55
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251297/

[7251299] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:26:38
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251299/

[7251301] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:23:15
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251301/

[7251305] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   01:19:24
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251305/

[7251307] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:31:05
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251307/

[7251309] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind}
time:   00:56:01
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251309/

[7251312] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:18:11
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251312/

[7251316] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:58
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251316/

[7251318] fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   01:09:50
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251318/

[7251320] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest}
time:   02:03:10
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251320/

[7251323] fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   01:12:25
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251323/

[7251325] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:19:46
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251325/

[7251329] fs/mirror-ha/{begin cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit}
time:   02:38:46
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251329/

[7251331] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:37:02
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251331/

[7251333] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:48:01
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251333/

[7251335] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:30:03
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251335/

[7251337] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   01:06:28
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251337/

[7251339] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:38:32
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251339/

[7251341] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:32:36
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251341/

[7251343] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop}
time:   01:03:14
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251343/

[7251345] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:22:43
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251345/

[7251347] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   01:51:26
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251347/

[7251351] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client}
time:   00:41:09
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251351/

[7251353] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   01:14:38
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251353/

[7251355] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}}
time:   00:58:25
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251355/

[7251359] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:15:21
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251359/

[7251361] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:15:24
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251361/

[7251363] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:14:51
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251363/

[7251365] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:44:41
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251365/

[7251369] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
time:   01:08:43
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251369/

[7251373] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:45:53
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251373/

[7251375] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}}
time:   01:27:31
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251375/

[7251377] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:16:39
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251377/

[7251379] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   01:09:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251379/

[7251381] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:22
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251381/

[7251383] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:13:35
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251383/

[7251385] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}}
time:   00:17:56
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251385/

[7251391] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:20:33
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251391/

[7251395] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   02:25:25
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251395/

[7251397] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:12:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251397/

[7251399] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
time:   01:41:20
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251399/

[7251401] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:13:20
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251401/

[7251403] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:23:08
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251403/

[7251405] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:54:25
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251405/

[7251407] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:43:36
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251407/

[7251409] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:53:27
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251409/

[7251411] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   01:10:58
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251411/

[7251415] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}}
time:   01:07:16
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251415/

[7251417] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:14:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251417/

[7251419] fs/upgrade/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04}
time:   01:13:18
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251419/

[7251421] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   02:21:22
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251421/

[7251423] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   01:11:19
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251423/

[7251425] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:11:30
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251425/

[7251427] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:12:54
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251427/

[7251429] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:21:06
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251429/

[7251433] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:18:39
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251433/

[7251435] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
time:   00:44:54
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251435/

[7251439] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   01:55:02
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251439/

[7251441] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   01:27:25
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251441/

[7251443] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:41:04
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251443/

[7251445] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:28:07
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251445/

[7251447] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:33:08
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251447/

[7251449] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:17:54
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251449/

[7251453] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}}
time:   01:23:26
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251453/

[7251455] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
time:   00:52:22
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251455/

[7251461] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   01:23:33
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251461/

[7251463] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:07
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251463/

[7251465] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   01:21:37
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251465/

[7251467] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:43
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251467/

[7251469] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:34:55
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251469/

[7251471] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:48:59
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251471/

[7251473] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:48:56
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251473/

[7251479] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}}
time:   00:28:47
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251479/

[7251480] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:48:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251480/

[7251482] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:20:37
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251482/

[7251484] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:14:33
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251484/

[7251488] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   01:14:00
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251488/

[7251492] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:44:33
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251492/

[7251494] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}}
time:   00:53:54
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251494/

[7251497] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:12:30
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251497/

[7251498] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   01:31:06
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251498/

[7251500] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   01:06:09
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251500/

[7251502] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:17:37
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251502/

[7251505] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:37:20
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251505/

[7251507] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:48:22
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251507/

[7251509] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:15
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251509/

[7251515] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:15:44
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251515/

[7251517] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:46:53
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251517/

[7251519] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
time:   00:44:09
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251519/

[7251525] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:17:39
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251525/

[7251527] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:51:15
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251527/

[7251529] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:08:22
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251529/

[7251531] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   01:09:20
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251531/

[7251533] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:19:13
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251533/

[7251537] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   01:47:38
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251537/

[7251539] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   01:27:04
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251539/

[7251541] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   01:08:50
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251541/

[7251543] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   01:09:19
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251543/

[7251547] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:31
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251547/

[7251549] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:56
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251549/

[7251551] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:14:05
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251551/

[7251553] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:45:39
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251553/

[7251557] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   00:17:43
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251557/

[7251559] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   01:32:58
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251559/

[7251561] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:12:36
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251561/

[7251563] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}}
time:   00:19:37
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251563/

[7251565] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:55:20
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251565/

[7251567] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:12:02
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251567/

[7251575] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:16:36
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251575/

[7251577] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}}
time:   01:22:06
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251577/

[7251581] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep}
time:   01:23:16
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251581/

[7251583] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:45
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251583/

[7251585] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   01:04:08
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251585/

[7251587] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:25:32
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251587/

[7251589] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   01:08:57
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251589/

[7251591] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:38:03
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251591/

[7251593] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:27:58
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251593/

[7251595] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client}
time:   00:30:29
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251595/

[7251599] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   01:54:31
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251599/

[7251603] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:12:38
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251603/

[7251605] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   01:08:04
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251605/

[7251607] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   01:20:45
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251607/

[7251609] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}}
time:   00:15:31
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251609/

[7251611] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:46:42
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251611/

[7251613] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:32:08
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251613/

[7251615] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:52:09
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251615/

[7251617] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:14:14
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251617/

[7251621] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:19:42
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251621/

[7251623] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:15:44
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251623/

[7251625] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   01:23:31
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251625/

[7251627] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:52
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251627/

[7251631] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:13:20
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251631/

[7251633] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:46:38
info:   http://pulpito.ceph.com/yuriw-2023-04-25_14:19:21-fs-pacific-release-distro-default-smithi/7251633/