2023-04-17T19:43:04.428 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-17T19:43:04.429 INFO:teuthology.results:subset: '111/12000'
2023-04-17T19:43:04.429 INFO:teuthology.results:seed: '2835'
2023-04-18T00:39:04.796 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-18T00:39:04.819 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-18T10:31:54.021 INFO:teuthology.results:Tests finished! gathering results...
2023-04-18T10:32:08.187 INFO:scrape:Found 172 jobs
2023-04-18T10:32:21.321 INFO:scrape:Found 12 distinct failure reasons
2023-04-18T10:32:21.322 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-04-18T10:32:21.322 INFO:scrape:2 jobs: ['7244466', '7244598']
2023-04-18T10:32:21.322 INFO:scrape:suites intersection: ['2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'thrashosds-health}']
2023-04-18T10:32:21.323 INFO:scrape:suites union: ['2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2023-04-18T10:32:21.323 INFO:scrape:
2023-04-18T10:32:21.323 INFO:scrape:Failure: more than one file system available
2023-04-18T10:32:21.324 INFO:scrape:14 jobs: ['7244524', '7244565', '7244595', '7244515', '7244609', '7244534', '7244618', '7244576', '7244456', '7244494', '7244586', '7244502', '7244487', '7244554']
2023-04-18T10:32:21.324 INFO:scrape:suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health}']
2023-04-18T10:32:21.324 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-04-18T10:32:21.325 INFO:scrape:
2023-04-18T10:32:21.325 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_sync (tasks.cephfs.test_mirroring.TestMirroring)
2023-04-18T10:32:21.325 INFO:scrape:2 jobs: ['7244483', '7244472']
2023-04-18T10:32:21.326 INFO:scrape:suites intersection: ['mount/fuse']
2023-04-18T10:32:21.326 INFO:scrape:suites union: ['centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/whitelist_health', 'overrides/{whitelist_health}', 'supported-random-distros$/{rhel_8}', 'tasks/mirror}', 'tasks/mirror}}']
2023-04-18T10:32:21.326 INFO:scrape:
2023-04-18T10:32:21.326 INFO:scrape:Failure: "2023-04-17T23:08:17.513538+0000 mon.a (mon.0) 1018 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2023-04-18T10:32:21.327 INFO:scrape:3 jobs: ['7244562', '7244607', '7244552']
2023-04-18T10:32:21.327 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-04-18T10:32:21.327 INFO:scrape:suites union: ['clusters/1-mds-2-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/functional/{begin', 'fs/multiclient/{begin', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}', 'tasks/snap-schedule}', 'tasks/volumes/{overrides', 'test/misc}}']
2023-04-18T10:32:21.328 INFO:scrape:
2023-04-18T10:32:21.328 INFO:scrape:Failure: {'smithi202.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 38.145.60.21)", 'rc': 1, 'results': []}, 'smithi141.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 34.221.3.152)", 'rc': 1, 'results': []}}
2023-04-18T10:32:21.328 INFO:scrape:4 jobs: ['7244464', '7244465', '7244461', '7244459']
2023-04-18T10:32:21.329 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-04-18T10:32:21.329 INFO:scrape:suites union: ['2-workunit/iozone}}', '2-workunit/suites/ffsb}}', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'fs/workload/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/yes', 'races', 'ranks/3', 'rhel_8}', 'scrub/no', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/test_journal_migration}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'workunit/fs/test_o_trunc}', 'wsync/{yes}}']
2023-04-18T10:32:21.329 INFO:scrape:
2023-04-18T10:32:21.330 INFO:scrape:Failure: Command failed on smithi142 with status 1: 'sudo yum install -y kernel'
2023-04-18T10:32:21.330 INFO:scrape:6 jobs: ['7244567', '7244546', '7244537', '7244463', '7244608', '7244625']
2023-04-18T10:32:21.330 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-04-18T10:32:21.331 INFO:scrape:suites union: ['1-client', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-upgrade', '2-workunit/cfuse_workunit_snaptests}}', '3-client-upgrade', '3-compat_client/pacific}}', '3-inline/yes', '3-upgrade-with-workload', '4-compat_client', '4-verify}', '4-verify}}', '5-client-sanity}}', 'centos_8.stream_container_tools', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'multimds/no', 'multimds/yes', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/1', 'roles', 'session_timeout', 'session_timeout}', 'tasks/dbench', 'tasks/{0-from/v16.2.4', 'tasks/{0-nautilus', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/valgrind}']
2023-04-18T10:32:21.331 INFO:scrape:
2023-04-18T10:32:21.331 INFO:scrape:Failure: {'smithi088.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/16bdfd9c9155376ddc8f552e5bcb7a6deb94739e35f439ba1a5db715bac5ddd9-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/25530bb7730289658577cd6a7aed7adb251d5fb40a42ebce2cb1f6d1cf1b1f87-filelists.xml.gz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}}
2023-04-18T10:32:21.332 INFO:scrape:2 jobs: ['7244468', '7244612']
2023-04-18T10:32:21.332 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health']
2023-04-18T10:32:21.332 INFO:scrape:suites union: ['centos_latest', 'clusters/1a3s-mds-4c-client', 'clusters/small-cluster', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/bugs/client_trim_caps/{begin', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/snapshots}', 'tasks/trim-i24137}']
2023-04-18T10:32:21.333 INFO:scrape:
2023-04-18T10:32:21.333 INFO:scrape:Failure: Test failure: test_delayed_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2023-04-18T10:32:21.333 INFO:scrape:1 jobs: ['7244570']
2023-04-18T10:32:21.333 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/metrics}']
2023-04-18T10:32:21.334 INFO:scrape:
2023-04-18T10:32:21.334 INFO:scrape:Failure: Command failed on smithi202 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e55275a6-dd8e-11ed-9b00-001a4aab830c -- ceph orch daemon add osd smithi202:vg_nvme/lv_3'
2023-04-18T10:32:21.334 INFO:scrape:1 jobs: ['7244614']
2023-04-18T10:32:21.335 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2023-04-18T10:32:21.335 INFO:scrape:
2023-04-18T10:32:21.335 INFO:scrape:Dead: b'2023-04-18T10:27:42.968 DEBUG:teuthology.exit:Finished running handlers'
2023-04-18T10:32:21.335 INFO:scrape:5 jobs: ['7244549', '7244527', '7244460', '7244529', '7244491']
2023-04-18T10:32:21.336 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-04-18T10:32:21.336 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/client-recovery}', 'tasks/{0-from/v16.2.4']
2023-04-18T10:32:21.336 INFO:scrape:
2023-04-18T10:32:21.337 INFO:scrape:Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)
2023-04-18T10:32:21.337 INFO:scrape:1 jobs: ['7244564']
2023-04-18T10:32:21.337 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/mds-full}']
2023-04-18T10:32:21.338 INFO:scrape:
2023-04-18T10:32:21.338 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2023-04-18T10:32:21.338 INFO:scrape:1 jobs: ['7244492']
2023-04-18T10:32:21.338 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}']
2023-04-18T10:32:21.339 INFO:scrape:
2023-04-18T10:32:21.348 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/
failed:  31
dead:    11
running: 0
waiting: 0
queued:  0
passed:  130


Fail
=================================================================
[7244456]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:17:03
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244456/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244456/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3d3e71cde9a34017aab282a8cdce1689

    more than one file system available

[7244463]  fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244463/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244463/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6f113ab7b9424f44b0b3bf0a5693d7fc

    Command failed on smithi153 with status 1: 'sudo yum install -y kernel'

[7244466]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   07:17:08
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244466/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244466/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1d723c67de8f422eb3d0baa8a555f43a

    Command failed (workunit test suites/fsstress.sh) on smithi057 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=70b1c583dc50212b0fe396879fa804459d145d7b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7244472]  fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror}
-----------------------------------------------------------------
time:   00:32:26
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244472/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244472/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a275852d5954406b8c3b36ed46769cda

    Test failure: test_cephfs_mirror_cancel_sync
    (tasks.cephfs.test_mirroring.TestMirroring)

[7244483]  fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   01:38:14
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244483/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244483/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f3f776b203f9438f8f8a7dbeaa7f7975

    Test failure: test_cephfs_mirror_cancel_sync
    (tasks.cephfs.test_mirroring.TestMirroring)

[7244487]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:20:20
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244487/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244487/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1cdd33abec034caeb7e638f1df2535f4

    more than one file system available

[7244492]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
-----------------------------------------------------------------
time:   00:15:11
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244492/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244492/
sentry: https://sentry.ceph.com/organizations/ceph/?query=962743410048449d9b71f80e33e2baef

    Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)

[7244494]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:15:39
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244494/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244494/
sentry: https://sentry.ceph.com/organizations/ceph/?query=54a34bec57c741c8bcfd9d31cf41dd8d

    more than one file system available

[7244502]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   01:20:38
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244502/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244502/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2596e39b787e4a4281e056087aaf5e05

    more than one file system available

[7244515]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:16:31
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244515/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244515/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd20a53cf7304b8085183938ec004c2d

    more than one file system available

[7244524]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:14:19
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244524/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244524/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ce8798b1f70d47a884541688fbbe4993

    more than one file system available

[7244534]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:16:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244534/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244534/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0295d0db6d3e4e94bda7193e7c2918b7

    more than one file system available

[7244537]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244537/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244537/
sentry: https://sentry.ceph.com/organizations/ceph/?query=08e4582854cc4a40aca4549c13e8d37a

    Command failed on smithi002 with status 1: 'sudo yum install -y kernel'

[7244546]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244546/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244546/
sentry: https://sentry.ceph.com/organizations/ceph/?query=182f52bbd4fb4689b93c90a092551a2e

    Command failed on smithi132 with status 1: 'sudo yum upgrade -y linux-
    firmware'

[7244552]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:22:33
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244552/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244552/

    "2023-04-17T22:51:52.272232+0000 mon.a (mon.0) 977 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7244554]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:16:55
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244554/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244554/
sentry: https://sentry.ceph.com/organizations/ceph/?query=076a9a8c9aee4d61bb8b6d7ec6f86c5c

    more than one file system available

[7244562]  fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:48:06
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244562/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244562/

    "2023-04-17T23:08:17.513538+0000 mon.a (mon.0) 1018 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7244564]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
-----------------------------------------------------------------
time:   01:21:05
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244564/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244564/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2fee1c3f75b848b08c8ecd404256b2bf

    Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)

[7244565]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:14:18
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244565/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244565/
sentry: https://sentry.ceph.com/organizations/ceph/?query=858cafea24ba454da9d551d2bfb1e6e9

    more than one file system available

[7244567]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244567/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244567/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4cc00a4593084b54ae3a2a52c8b638ab

    Command failed on smithi142 with status 1: 'sudo yum install -y kernel'

[7244570]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   01:12:38
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244570/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244570/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5f1dbaff29f7457ab0ca343ea279979d

    Test failure: test_delayed_metrics
    (tasks.cephfs.test_mds_metrics.TestMDSMetrics)

[7244576]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:15:27
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244576/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244576/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b160ff932491440a89d6bf38c894e413

    more than one file system available

[7244586]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   01:18:26
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244586/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244586/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d50a56a8e3e14aadab5dd0af712d9892

    more than one file system available

[7244595]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:16:38
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244595/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244595/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6f4d359af8b743baaa81f422fc91fd28

    more than one file system available

[7244598]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   06:55:40
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244598/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244598/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7d0e1e6811774bb9a8c47213acbe5e1d

    Command failed (workunit test suites/fsstress.sh) on smithi019 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=70b1c583dc50212b0fe396879fa804459d145d7b
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7244607]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   01:43:04
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244607/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244607/

    "2023-04-18T01:43:46.730072+0000 mon.a (mon.0) 435 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7244608]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244608/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244608/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e4e0d7c1edb14825bd848b2dff9a0fcf

    Command failed on smithi047 with status 1: 'sudo yum install -y kernel'

[7244609]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:16:52
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244609/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244609/
sentry: https://sentry.ceph.com/organizations/ceph/?query=271b8dc8634a441cb4b42566e7b71337

    more than one file system available

[7244614]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   01:33:42
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244614/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244614/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e951aba9ac94b43b41ef6877c119ff2

    Command failed on smithi202 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e55275a6-dd8e-11ed-9b00-001a4aab830c -- ceph orch daemon add osd
    smithi202:vg_nvme/lv_3'

[7244618]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:14:43
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244618/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244618/
sentry: https://sentry.ceph.com/organizations/ceph/?query=470d1ec8f599415e9ab14f4585c89d91

    more than one file system available

[7244625]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244625/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244625/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ab47dc04ff1a4ddd80d9b9ea9fe1fa0d

    Command failed on smithi181 with status 1: 'sudo yum install -y kernel'



Dead
=================================================================
[7244459]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:04:15
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244459/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244459/
sentry: https://sentry.ceph.com/organizations/ceph/?query=540db768c417435d9203b4a93d4b0dfc

    {'smithi141.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}, 'smithi202.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}}

[7244460]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244460/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244460/

    hit max job timeout

[7244461]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
-----------------------------------------------------------------
time:   00:04:16
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244461/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244461/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0c25daa708aa4ae88256dcc31cd9908f

    {'smithi088.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []},
    'smithi203.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}, 'smithi158.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []}}

[7244464]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:04:11
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244464/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244464/
sentry: https://sentry.ceph.com/organizations/ceph/?query=92a148d4725b4c8d9256a9d84bcbff7b

    {'smithi202.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}, 'smithi141.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}}

[7244465]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:04:13
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244465/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244465/
sentry: https://sentry.ceph.com/organizations/ceph/?query=125b37f336b74aff8916a1ef4c6ae4ac

    {'smithi072.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}, 'smithi117.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []}}

[7244468]  fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
-----------------------------------------------------------------
time:   00:21:27
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244468/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244468/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6625a58a337e4adf87c84efacb4c8619

    {'smithi088.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/16bdfd9c9155376d
    dc8f552e5bcb7a6deb94739e35f439ba1a5db715bac5ddd9-primary.xml.gz - Cannot
    download, all mirrors were already tried without success; repodata/25530bb7
    730289658577cd6a7aed7adb251d5fb40a42ebce2cb1f6d1cf1b1f87-filelists.xml.gz -
    Cannot download, all mirrors were already tried without success", 'rc': 1,
    'results': []}}

[7244491]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244491/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244491/

    hit max job timeout

[7244527]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244527/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244527/

    hit max job timeout

[7244529]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244529/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244529/

    hit max job timeout

[7244549]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244549/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244549/

    hit max job timeout

[7244612]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
-----------------------------------------------------------------
time:   00:40:50
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244612/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244612/
sentry: https://sentry.ceph.com/organizations/ceph/?query=36fdf65c4a4d46e393555f1a4529c014

    {'smithi103.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/25530bb773028965
    8577cd6a7aed7adb251d5fb40a42ebce2cb1f6d1cf1b1f87-filelists.xml.gz - Cannot
    download, all mirrors were already tried without success", 'rc': 1,
    'results': []}}



Pass
=================================================================
[7244454] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:54:01
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244454/

[7244455] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
time:   00:16:27
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244455/

[7244457] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   01:14:03
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244457/

[7244458] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:27:34
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244458/

[7244462] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   01:26:21
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244462/

[7244467] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:06
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244467/

[7244469] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest}
time:   02:07:10
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244469/

[7244470] fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:19:39
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244470/

[7244471] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:14:53
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244471/

[7244473] fs/mirror-ha/{begin cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit}
time:   02:35:45
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244473/

[7244474] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:39:02
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244474/

[7244475] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:48:24
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244475/

[7244476] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   01:25:46
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244476/

[7244477] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:40
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244477/

[7244478] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:38:24
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244478/

[7244479] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:42:28
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244479/

[7244480] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop}
time:   01:05:31
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244480/

[7244481] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:20:03
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244481/

[7244482] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   01:53:46
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244482/

[7244484] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client}
time:   00:32:27
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244484/

[7244485] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   01:23:06
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244485/

[7244486] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}}
time:   00:52:41
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244486/

[7244488] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:15:19
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244488/

[7244489] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   01:12:29
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244489/

[7244490] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:14:27
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244490/

[7244493] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}}
time:   01:15:58
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244493/

[7244495] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:53:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244495/

[7244496] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}}
time:   01:32:02
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244496/

[7244497] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   01:11:54
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244497/

[7244498] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:54
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244498/

[7244499] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   01:11:02
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244499/

[7244500] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:13:48
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244500/

[7244501] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
time:   00:17:49
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244501/

[7244503] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:30:56
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244503/

[7244504] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   01:18:02
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244504/

[7244505] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep}
time:   00:22:51
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244505/

[7244506] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   02:31:27
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244506/

[7244507] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:12:08
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244507/

[7244508] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
time:   01:47:39
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244508/

[7244509] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:17:42
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244509/

[7244510] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:16:51
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244510/

[7244511] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:54:01
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244511/

[7244512] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:47:35
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244512/

[7244513] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   01:02:28
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244513/

[7244514] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   01:11:37
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244514/

[7244516] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}}
time:   00:22:54
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244516/

[7244517] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:13:32
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244517/

[7244518] fs/upgrade/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04}
time:   01:14:34
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244518/

[7244519] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   02:27:28
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244519/

[7244520] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   01:18:42
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244520/

[7244521] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   01:11:01
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244521/

[7244522] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:35
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244522/

[7244523] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   01:16:20
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244523/

[7244525] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:12:34
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244525/

[7244526] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
time:   00:42:20
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244526/

[7244528] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   02:03:20
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244528/

[7244530] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:45:05
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244530/

[7244531] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:27:45
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244531/

[7244532] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:31:51
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244532/

[7244533] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:19:00
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244533/

[7244535] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
time:   01:31:38
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244535/

[7244536] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
time:   00:52:23
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244536/

[7244538] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   09:24:43
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244538/

[7244539] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   01:22:32
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244539/

[7244540] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   01:13:11
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244540/

[7244541] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   01:21:20
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244541/

[7244542] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:13
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244542/

[7244543] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:43:23
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244543/

[7244544] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:46:58
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244544/

[7244545] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:44:23
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244545/

[7244547] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   01:15:49
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244547/

[7244548] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
time:   00:29:46
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244548/

[7244550] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:21:27
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244550/

[7244551] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:13:07
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244551/

[7244553] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   01:14:31
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244553/

[7244555] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:46:51
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244555/

[7244556] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}}
time:   01:46:31
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244556/

[7244557] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:14:28
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244557/

[7244558] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   01:28:10
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244558/

[7244559] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:23:49
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244559/

[7244560] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:16:18
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244560/

[7244561] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:38:36
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244561/

[7244563] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:39
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244563/

[7244566] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:16:28
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244566/

[7244568] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}}
time:   00:49:56
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244568/

[7244569] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
time:   01:39:44
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244569/

[7244571] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:18:03
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244571/

[7244572] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:51:58
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244572/

[7244573] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:03:10
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244573/

[7244574] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   01:15:22
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244574/

[7244575] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:19:11
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244575/

[7244577] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   01:49:12
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244577/

[7244578] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   01:30:04
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244578/

[7244579] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   01:13:04
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244579/

[7244580] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
time:   01:20:07
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244580/

[7244581] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:48:24
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244581/

[7244582] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:22:28
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244582/

[7244583] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:14:08
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244583/

[7244584] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:14:10
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244584/

[7244585] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:56:02
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244585/

[7244587] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   00:15:34
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244587/

[7244588] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   01:32:11
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244588/

[7244589] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:14:03
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244589/

[7244590] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}}
time:   00:19:53
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244590/

[7244591] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   01:18:51
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244591/

[7244592] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:11:55
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244592/

[7244593] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   01:26:07
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244593/

[7244594] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:51:46
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244594/

[7244596] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:16:38
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244596/

[7244597] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:27:52
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244597/

[7244599] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep}
time:   02:21:50
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244599/

[7244600] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:05
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244600/

[7244601] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:13:06
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244601/

[7244602] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:24:59
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244602/

[7244603] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:55
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244603/

[7244604] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:40:42
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244604/

[7244605] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:23:41
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244605/

[7244606] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client}
time:   00:27:56
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244606/

[7244610] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:12:15
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244610/

[7244611] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   01:13:52
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244611/

[7244613] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}}
time:   00:19:39
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244613/

[7244615] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:31:45
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244615/

[7244616] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:25:32
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244616/

[7244617] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:14:10
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244617/

[7244619] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:18:46
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244619/

[7244620] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:16:07
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244620/

[7244621] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   01:25:50
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244621/

[7244622] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:12
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244622/

[7244623] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   01:16:23
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244623/

[7244624] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:13:05
info:   http://pulpito.ceph.com/yuriw-2023-04-17_15:37:37-fs-wip-yuri2-testing-2023-04-12-0728-pacific-distro-default-smithi/7244624/