2023-04-23T20:20:00.979 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-23T20:20:00.981 INFO:teuthology.results:subset: '111/12000'
2023-04-23T20:20:00.981 INFO:teuthology.results:seed: '2397'
2023-04-24T00:34:59.303 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-24T00:34:59.359 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-24T12:28:16.830 INFO:teuthology.results:Tests finished! gathering results...
2023-04-24T12:28:31.445 INFO:scrape:Found 172 jobs
2023-04-24T12:28:39.981 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248752/teuthology.log
2023-04-24T12:28:46.531 INFO:scrape:Found 12 distinct failure reasons
2023-04-24T12:28:46.532 INFO:scrape:Failure: more than one file system available
2023-04-24T12:28:46.533 INFO:scrape:16 jobs: ['7248800', '7248788', '7248769', '7248748', '7248819', '7248849', '7248718', '7248710', '7248863', '7248840', '7248808', '7248756', '7248830', '7248872', '7248741', '7248778']
2023-04-24T12:28:46.533 INFO:scrape:suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health}']
2023-04-24T12:28:46.533 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-04-24T12:28:46.534 INFO:scrape:
2023-04-24T12:28:46.534 INFO:scrape:Failure: {'smithi188.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/144bb6d03f4cceafda5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}, 'smithi142.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/655edd281b923f12af44ba71f23c34741aa8dae0a29e712e75d96be8787f7115-modules.yaml.xz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}}
2023-04-24T12:28:46.535 INFO:scrape:3 jobs: ['7248876', '7248833', '7248783']
2023-04-24T12:28:46.535 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2023-04-24T12:28:46.535 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/32bits/{begin', 'fs/functional/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/client-recovery}', 'tasks/openfiletable}']
2023-04-24T12:28:46.536 INFO:scrape:
2023-04-24T12:28:46.536 INFO:scrape:Dead: b'2023-04-24T10:07:26.382 DEBUG:teuthology.exit:Finished running handlers'
2023-04-24T12:28:46.536 INFO:scrape:3 jobs: ['7248781', '7248868', '7248714']
2023-04-24T12:28:46.536 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2023-04-24T12:28:46.537 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2023-04-24T12:28:46.537 INFO:scrape:
2023-04-24T12:28:46.537 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2023-04-24T12:28:46.538 INFO:scrape:1 jobs: ['7248746']
2023-04-24T12:28:46.538 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}']
2023-04-24T12:28:46.538 INFO:scrape:
2023-04-24T12:28:46.538 INFO:scrape:Failure: Command failed on smithi050 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd'
2023-04-24T12:28:46.539 INFO:scrape:1 jobs: ['7248736']
2023-04-24T12:28:46.539 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/no}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus']
2023-04-24T12:28:46.539 INFO:scrape:
2023-04-24T12:28:46.540 INFO:scrape:Failure: "2023-04-23T21:47:51.094632+0000 mon.a (mon.0) 300 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
2023-04-24T12:28:46.540 INFO:scrape:3 jobs: ['7248755', '7248861', '7248843']
2023-04-24T12:28:46.540 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2023-04-24T12:28:46.541 INFO:scrape:suites union: ['clusters/1-mds-3-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distros/ubuntu_latest', 'fs/functional/{begin', 'fs/multiclient/{begin', 'fs/workload/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'overrides/{ignorelist_health', 'ranks/3', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/ior-shared-file}', 'tasks/snap-schedule}', 'tasks/{0-check-counter', 'workunit/suites/pjd}', 'wsync/{yes}}']
2023-04-24T12:28:46.541 INFO:scrape:
2023-04-24T12:28:46.541 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_sync (tasks.cephfs.test_mirroring.TestMirroring)
2023-04-24T12:28:46.541 INFO:scrape:2 jobs: ['7248737', '7248726']
2023-04-24T12:28:46.542 INFO:scrape:suites intersection: ['mount/fuse']
2023-04-24T12:28:46.542 INFO:scrape:suites union: ['centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/whitelist_health', 'overrides/{whitelist_health}', 'supported-random-distros$/{centos_8}', 'tasks/mirror}', 'tasks/mirror}}']
2023-04-24T12:28:46.542 INFO:scrape:
2023-04-24T12:28:46.543 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-04-24T12:28:46.543 INFO:scrape:1 jobs: ['7248720']
2023-04-24T12:28:46.543 INFO:scrape:suites: ['2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-04-24T12:28:46.543 INFO:scrape:
2023-04-24T12:28:46.544 INFO:scrape:Dead: None
2023-04-24T12:28:46.544 INFO:scrape:1 jobs: ['7248752']
2023-04-24T12:28:46.544 INFO:scrape:
2023-04-24T12:28:46.545 INFO:scrape:Failure: Test failure: test_delayed_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2023-04-24T12:28:46.545 INFO:scrape:1 jobs: ['7248824']
2023-04-24T12:28:46.545 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/metrics}']
2023-04-24T12:28:46.545 INFO:scrape:
2023-04-24T12:28:46.546 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi083 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad9576de6cbdb0047be604610f5b00c42ad65335 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-04-24T12:28:46.546 INFO:scrape:1 jobs: ['7248835']
2023-04-24T12:28:46.546 INFO:scrape:suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-04-24T12:28:46.547 INFO:scrape:
2023-04-24T12:28:46.547 INFO:scrape:Failure: 
2023-04-24T12:28:46.547 INFO:scrape:1 jobs: ['7248709']
2023-04-24T12:28:46.547 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'ranks/1', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/fsync-tester}', 'wsync/{yes}}']
2023-04-24T12:28:46.548 INFO:scrape:
2023-04-24T12:28:46.557 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/
failed:  27
dead:    7
running: 0
waiting: 0
queued:  0
passed:  138


Fail
=================================================================
[7248709]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:26
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248709/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248709/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c585832520b84840969bb378cf30acd5

[7248710]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:16:34
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248710/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248710/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d3aca8229da64c669dac15eac34004c5

    more than one file system available

[7248718]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:14:14
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248718/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248718/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a44376dce60d4eda8f00c1678151fcf6

    more than one file system available

[7248720]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   07:19:11
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248720/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248720/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9fe0681dacca4372b8fc131223420a3f

    Command failed (workunit test suites/fsstress.sh) on smithi136 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ad9576de6cbdb0047be604610f5b00c42ad65335
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7248726]  fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror}
-----------------------------------------------------------------
time:   01:25:10
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248726/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248726/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6795385f730044529d07c472dab82770

    Test failure: test_cephfs_mirror_cancel_sync
    (tasks.cephfs.test_mirroring.TestMirroring)

[7248736]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   01:42:41
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248736/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248736/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b0661e47b3b94ae48fbce6554c1925a2

    Command failed on smithi050 with status 1: 'sudo yum -y upgrade ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2
    librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-
    mirror rbd-nbd'

[7248737]  fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   01:35:28
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248737/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248737/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4d355a5bfb184656a2547d0f2ef514e8

    Test failure: test_cephfs_mirror_cancel_sync
    (tasks.cephfs.test_mirroring.TestMirroring)

[7248741]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:17:57
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248741/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248741/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8da0e99b8b2b4eda810bc46bdd194e9e

    more than one file system available

[7248746]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
-----------------------------------------------------------------
time:   00:17:26
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248746/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248746/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8ca125c393bd4292aa903840e81375bb

    Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)

[7248748]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:19:40
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248748/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248748/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8d12c9107f664e14abcf1d45f7cd97de

    more than one file system available

[7248755]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}}
-----------------------------------------------------------------
time:   00:17:28
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248755/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248755/

    "2023-04-23T21:47:51.094632+0000 mon.a (mon.0) 300 : cluster [WRN] Health
    check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)"
    in cluster log

[7248756]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   01:11:43
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248756/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248756/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9fb1c0c80a7d4e47af0ff40af10ae0a9

    more than one file system available

[7248769]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:17:01
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248769/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248769/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a1b2b419f6d14b0e91106cf165e23eee

    more than one file system available

[7248778]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:14:26
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248778/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248778/
sentry: https://sentry.ceph.com/organizations/ceph/?query=80cbc049b6744a3e98daffb058e3b1e1

    more than one file system available

[7248788]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:22:44
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248788/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248788/
sentry: https://sentry.ceph.com/organizations/ceph/?query=67219ee53a5c49c4953ae340c4848ba7

    more than one file system available

[7248800]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   01:12:34
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248800/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248800/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cb92214c4e6b4e16804f36766f427d0b

    more than one file system available

[7248808]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:16:48
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248808/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248808/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca3bfbda12b44c1ea25c5f3d5784c829

    more than one file system available

[7248819]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:17:20
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248819/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248819/
sentry: https://sentry.ceph.com/organizations/ceph/?query=97cf546b6f714d92b309e1a4015e7469

    more than one file system available

[7248824]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:17:00
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248824/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248824/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3810bf237f734fba990ab7cbc276a047

    Test failure: test_delayed_metrics
    (tasks.cephfs.test_mds_metrics.TestMDSMetrics)

[7248830]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:15:14
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248830/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248830/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a61c0a3fe11e4923b3aa95ccccd544ac

    more than one file system available

[7248835]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:33:51
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248835/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248835/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a42ba4a765b45ccb58e19d58563872c

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi083
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ad9576de6cbdb0047be604610f5b00c42ad65335
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7248840]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   01:11:45
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248840/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248840/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3b76bb26be274f4b8ad7e0b0b99fc78a

    more than one file system available

[7248843]  fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:11:59
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248843/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248843/

    "2023-04-23T23:58:02.118765+0000 mon.a (mon.0) 179 : cluster [WRN] Health
    check failed: Degraded data redundancy: 1/46 objects degraded (2.174%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[7248849]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:16:22
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248849/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248849/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7cec9ca5499745d1a05c61530ade93fe

    more than one file system available

[7248861]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   01:42:47
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248861/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248861/

    "2023-04-24T01:47:09.590755+0000 mon.a (mon.0) 411 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7248863]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:16:56
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248863/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248863/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a99bce7308194c8b806c6f3e5175cc2c

    more than one file system available

[7248872]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:17:55
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248872/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248872/
sentry: https://sentry.ceph.com/organizations/ceph/?query=841ec46d3e2549fcab64bd8f3e88ae75

    more than one file system available



Dead
=================================================================
[7248714]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248714/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248714/

    hit max job timeout

[7248752]  fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248752/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248752/

    Error reimaging machines: 'NoneType' object has no attribute '_fields'

[7248781]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248781/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248781/

    hit max job timeout

[7248783]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:38:50
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248783/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248783/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e9428c9a99b641b1b91ae256396352e1

    {'smithi130.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s):
    repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-
    comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already
    tried without success; repodata/655edd281b923f12af44ba71f23c34741aa8dae0a29
    e712e75d96be8787f7115-modules.yaml.xz - Cannot download, all mirrors were
    already tried without success", 'rc': 1, 'results': []}}

[7248833]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
-----------------------------------------------------------------
time:   00:38:49
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248833/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248833/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b910546051284ad190821f3dbda4191c

    {'smithi181.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/144bb6d03f4cceaf
    da5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot
    download, all mirrors were already tried without success;
    repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-
    comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already
    tried without success", 'rc': 1, 'results': []}}

[7248868]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248868/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248868/

    hit max job timeout

[7248876]  fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:23:46
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248876/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248876/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d8ae8ab51c53479baed0b412905382f6

    {'smithi188.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/144bb6d03f4cceaf
    da5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot
    download, all mirrors were already tried without success;
    repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-
    comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already
    tried without success", 'rc': 1, 'results': []},
    'smithi142.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/655edd281b923f12
    af44ba71f23c34741aa8dae0a29e712e75d96be8787f7115-modules.yaml.xz - Cannot
    download, all mirrors were already tried without success", 'rc': 1,
    'results': []}}



Pass
=================================================================
[7248708] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:52:19
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248708/

[7248711] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   01:11:14
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248711/

[7248712] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:26:54
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248712/

[7248713] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:22:22
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248713/

[7248715] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   01:16:50
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248715/

[7248716] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:29:11
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248716/

[7248717] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind}
time:   01:51:30
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248717/

[7248719] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:17:36
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248719/

[7248721] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:15:45
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248721/

[7248722] fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   01:08:34
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248722/

[7248723] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest}
time:   02:09:22
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248723/

[7248724] fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:19:16
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248724/

[7248725] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:11:50
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248725/

[7248727] fs/mirror-ha/{begin cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit}
time:   03:27:44
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248727/

[7248728] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:39:55
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248728/

[7248729] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:48:29
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248729/

[7248730] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:31:23
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248730/

[7248731] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:35
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248731/

[7248732] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:38:45
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248732/

[7248733] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:48:35
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248733/

[7248734] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop}
time:   00:10:12
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248734/

[7248735] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   01:13:48
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248735/

[7248738] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client}
time:   01:29:25
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248738/

[7248739] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   01:12:28
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248739/

[7248740] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}}
time:   00:51:38
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248740/

[7248742] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:15:42
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248742/

[7248743] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   01:07:41
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248743/

[7248744] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:14:31
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248744/

[7248745] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:55:57
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248745/

[7248747] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}}
time:   01:10:53
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248747/

[7248749] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:53:30
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248749/

[7248750] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}}
time:   01:32:03
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248750/

[7248751] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:16:34
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248751/

[7248753] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:16
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248753/

[7248754] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:13:46
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248754/

[7248757] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:28:45
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248757/

[7248758] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:20:39
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248758/

[7248759] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep}
time:   00:19:15
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248759/

[7248760] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   02:28:06
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248760/

[7248761] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:14:16
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248761/

[7248762] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
time:   01:50:27
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248762/

[7248763] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:13:22
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248763/

[7248764] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:18:31
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248764/

[7248765] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:56:38
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248765/

[7248766] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:50:55
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248766/

[7248767] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   01:58:57
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248767/

[7248768] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   01:13:07
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248768/

[7248770] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}}
time:   00:17:30
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248770/

[7248771] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:13:33
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248771/

[7248772] fs/upgrade/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04}
time:   01:14:22
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248772/

[7248773] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   02:25:31
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248773/

[7248774] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:25:44
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248774/

[7248775] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   01:06:17
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248775/

[7248776] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:35
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248776/

[7248777] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   01:19:57
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248777/

[7248779] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:11:59
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248779/

[7248780] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}}
time:   00:38:18
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248780/

[7248782] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   02:04:29
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248782/

[7248784] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:32:57
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248784/

[7248785] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:28:05
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248785/

[7248786] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:33:29
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248786/

[7248787] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:21:29
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248787/

[7248789] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
time:   01:27:24
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248789/

[7248790] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
time:   00:52:23
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248790/

[7248791] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   02:03:29
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248791/

[7248792] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   01:24:23
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248792/

[7248793] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   01:23:18
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248793/

[7248794] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:18
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248794/

[7248795] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   01:29:44
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248795/

[7248796] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:19
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248796/

[7248797] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:38:13
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248797/

[7248798] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:46:23
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248798/

[7248799] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:43:44
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248799/

[7248801] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   01:15:35
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248801/

[7248802] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
time:   00:28:14
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248802/

[7248803] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:49:18
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248803/

[7248804] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:20:59
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248804/

[7248805] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:13:22
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248805/

[7248806] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:22:25
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248806/

[7248807] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:25:42
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248807/

[7248809] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:58:07
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248809/

[7248810] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}}
time:   00:49:33
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248810/

[7248811] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:12:18
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248811/

[7248812] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   01:24:22
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248812/

[7248813] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:49
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248813/

[7248814] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   01:10:22
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248814/

[7248815] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:39:30
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248815/

[7248816] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:48:18
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248816/

[7248817] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   01:15:11
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248817/

[7248818] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   01:28:20
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248818/

[7248820] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:15:59
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248820/

[7248821] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:47:52
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248821/

[7248822] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
time:   00:41:40
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248822/

[7248823] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
time:   01:36:10
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248823/

[7248825] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:20:56
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248825/

[7248826] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:52:34
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248826/

[7248827] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:02:19
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248827/

[7248828] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   01:11:51
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248828/

[7248829] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:18:57
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248829/

[7248831] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   01:51:58
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248831/

[7248832] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:37:32
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248832/

[7248834] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   01:15:52
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248834/

[7248836] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:18:43
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248836/

[7248837] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:15:54
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248837/

[7248838] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:13:58
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248838/

[7248839] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:45:20
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248839/

[7248841] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   01:09:28
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248841/

[7248842] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   01:33:01
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248842/

[7248844] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}}
time:   00:19:42
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248844/

[7248845] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:58:34
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248845/

[7248846] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:16:59
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248846/

[7248847] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   01:19:35
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248847/

[7248848] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:47:42
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248848/

[7248850] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:16:51
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248850/

[7248851] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:23:33
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248851/

[7248852] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:24:58
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248852/

[7248853] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep}
time:   01:10:14
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248853/

[7248854] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:15
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248854/

[7248855] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:11:58
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248855/

[7248856] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:25:50
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248856/

[7248857] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   01:13:05
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248857/

[7248858] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:38:01
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248858/

[7248859] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   01:23:48
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248859/

[7248860] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client}
time:   00:28:25
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248860/

[7248862] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   02:02:49
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248862/

[7248864] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:12:02
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248864/

[7248865] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   01:10:22
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248865/

[7248866] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   01:22:40
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248866/

[7248867] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
time:   00:21:31
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248867/

[7248869] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:32:27
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248869/

[7248870] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:52:59
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248870/

[7248871] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:14:49
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248871/

[7248873] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:18:10
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248873/

[7248874] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:16:40
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248874/

[7248875] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
time:   01:23:50
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248875/

[7248877] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   01:17:43
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248877/

[7248878] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:17:44
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248878/

[7248879] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:49:24
info:   http://pulpito.ceph.com/yuriw-2023-04-23_18:53:08-fs-pacific-distro-default-smithi/7248879/