2023-04-14T21:30:17.933 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-14T21:30:17.934 INFO:teuthology.results:subset: '111/12000'
2023-04-14T21:30:17.934 INFO:teuthology.results:seed: '618'
2023-04-15T01:09:30.299 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-15T01:09:30.323 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-15T13:09:34.461 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-04-15T13:09:50.378 INFO:scrape:Found 172 jobs
2023-04-15T13:10:01.405 INFO:scrape:Found 9 distinct failure reasons
2023-04-15T13:10:01.406 INFO:scrape:Failure: {'smithi084.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 8.43.85.73)", 'rc': 1, 'results': []}, 'smithi122.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 8.43.85.73)", 'rc': 1, 'results': []}}
2023-04-15T13:10:01.407 INFO:scrape:47 jobs
2023-04-15T13:10:01.408 INFO:scrape:suites intersection: []
2023-04-15T13:10:01.408 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/trivial_sync}}', '2-workunit/iozone}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', 'cluster/{1-node}', 'clusters/1-node-1-mds-1-osd', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/32bits/{begin', 'fs/full/{begin', 'fs/functional/{begin', 'fs/mixed-clients/{begin', 'fs/multifs/{begin', 'fs/permission/{begin', 'fs/snaps/{begin', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'fs/top/{begin', 'fs/traceless/{begin', 'fs/verify/{begin', 'fs/volumes/{begin', 'fs/workload/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'k-testing}', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'mon-debug', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides', 'overrides/ignorelist_health', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/no', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'supported-random-distros$/{rhel_8}', 'tasks/auto-repair}', 'tasks/backtrace}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-readahead}', 'tasks/dbench', 'tasks/exports}', 'tasks/failover}', 'tasks/fsstress', 'tasks/fstop}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/mds-flush}', 'tasks/metrics}', 'tasks/mgr-osd-full}', 'tasks/multimds_misc}', 'tasks/recovery-fs}', 'tasks/snap-schedule}', 'tasks/snap_schedule_snapdir}', 'tasks/test_journal_migration}', 'tasks/truncate_delay}', 'tasks/volume-client}', 'tasks/volumes/{overrides', 'tasks/workunit/quota}', 'tasks/workunit/snaps}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'test/basic}}', 'thrashosds-health}', 'traceless/50pc}', 'validater/lockdep}', 'validater/valgrind}', 'workunit/fs/misc}', 'workunit/suites/dbench}', 'workunit/suites/ffsb}', 'workunit/suites/fsx}', 'workunit/suites/fsync-tester}', 'workunit/suites/iogen}', 'workunit/suites/iozone}', 'wsync/{no}}', 'wsync/{yes}}']
2023-04-15T13:10:01.409 INFO:scrape:
2023-04-15T13:10:01.409 INFO:scrape:Failure: {'smithi142.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 209.132.190.2)", 'rc': 1, 'results': []}, 'smithi131.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 8.43.85.67)", 'rc': 1, 'results': []}}
2023-04-15T13:10:01.409 INFO:scrape:6 jobs: ['7241446', '7241552', '7241517', '7241458', '7241435', '7241559']
2023-04-15T13:10:01.410 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2023-04-15T13:10:01.410 INFO:scrape:suites union: ['2-workunit/suites/fsstress}}', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/permission/{begin', 'fs/thrash/workloads/{begin', 'fs/traceless/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/acls}', 'tasks/asok_dump_tree}', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'traceless/50pc}']
2023-04-15T13:10:01.410 INFO:scrape:
2023-04-15T13:10:01.411 INFO:scrape:Failure: more than one file system available
2023-04-15T13:10:01.411 INFO:scrape:11 jobs: ['7241508', '7241500', '7241540', '7241572', '7241519', '7241549', '7241418', '7241478', '7241410', '7241563', '7241469']
2023-04-15T13:10:01.411 INFO:scrape:suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health}']
2023-04-15T13:10:01.412 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-04-15T13:10:01.412 INFO:scrape:
2023-04-15T13:10:01.412 INFO:scrape:Dead: b'2023-04-15T11:39:11.715 DEBUG:teuthology.exit:Finished running handlers'
2023-04-15T13:10:01.413 INFO:scrape:2 jobs: ['7241503', '7241568']
2023-04-15T13:10:01.413 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2023-04-15T13:10:01.413 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2023-04-15T13:10:01.413 INFO:scrape:
2023-04-15T13:10:01.414 INFO:scrape:Failure: "2023-04-15T01:00:03.632473+0000 mon.a (mon.0) 1673 : cluster [WRN] Health check failed: Degraded data redundancy: 1/46 objects degraded (2.174%), 1 pg degraded (PG_DEGRADED)" in cluster log
2023-04-15T13:10:01.414 INFO:scrape:1 jobs: ['7241482']
2023-04-15T13:10:01.414 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}']
2023-04-15T13:10:01.415 INFO:scrape:
2023-04-15T13:10:01.415 INFO:scrape:Failure: Command failed on smithi142 with status 1: 'sudo yum upgrade -y linux-firmware'
2023-04-15T13:10:01.415 INFO:scrape:1 jobs: ['7241456']
2023-04-15T13:10:01.416 INFO:scrape:suites: ['2-workunit/cfuse_workunit_suites_pjd}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-04-15T13:10:01.416 INFO:scrape:
2023-04-15T13:10:01.416 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_sync (tasks.cephfs.test_mirroring.TestMirroring)
2023-04-15T13:10:01.416 INFO:scrape:2 jobs: ['7241437', '7241426']
2023-04-15T13:10:01.417 INFO:scrape:suites intersection: ['mount/fuse']
2023-04-15T13:10:01.417 INFO:scrape:suites union: ['centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/whitelist_health', 'overrides/{whitelist_health}', 'supported-random-distros$/{centos_8}', 'tasks/mirror}', 'tasks/mirror}}']
2023-04-15T13:10:01.417 INFO:scrape:
2023-04-15T13:10:01.418 INFO:scrape:Failure: {'smithi145.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 8.43.85.67)", 'rc': 1, 'results': []}, 'smithi151.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP: 8.43.85.73)", 'rc': 1, 'results': []}}
2023-04-15T13:10:01.418 INFO:scrape:1 jobs: ['7241421']
2023-04-15T13:10:01.418 INFO:scrape:suites: ['clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{rhel_8}', 'fs/32bits/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{faked-ino', 'tasks/cfuse_workunit_suites_fsstress}']
2023-04-15T13:10:01.419 INFO:scrape:
2023-04-15T13:10:01.419 INFO:scrape:Failure: {'smithi049.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/16bdfd9c9155376ddc8f552e5bcb7a6deb94739e35f439ba1a5db715bac5ddd9-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/25530bb7730289658577cd6a7aed7adb251d5fb40a42ebce2cb1f6d1cf1b1f87-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}}
2023-04-15T13:10:01.419 INFO:scrape:1 jobs: ['7241509']
2023-04-15T13:10:01.420 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus']
2023-04-15T13:10:01.420 INFO:scrape:
2023-04-15T13:10:01.430 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/
failed:  15
dead:    56
running: 1
waiting: 0
queued:  0
passed:  100


Fail
=================================================================
[7241410]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:16:07
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241410/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241410/
sentry: https://sentry.ceph.com/organizations/ceph/?query=69318acd897c4843b3a41ae2d22b632f

    more than one file system available

[7241418]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:14:15
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241418/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241418/
sentry: https://sentry.ceph.com/organizations/ceph/?query=904c0d7ac0104a2ba3e3b970fb0932e8

    more than one file system available

[7241426]  fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror}
-----------------------------------------------------------------
time:   01:32:10
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241426/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241426/
sentry: https://sentry.ceph.com/organizations/ceph/?query=be5e0ee15162447a9169c90bb584a0fa

    Test failure: test_cephfs_mirror_cancel_sync
    (tasks.cephfs.test_mirroring.TestMirroring)

[7241437]  fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   01:42:52
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241437/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241437/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eaf6906846d548988079fdca771d88a6

    Test failure: test_cephfs_mirror_cancel_sync
    (tasks.cephfs.test_mirroring.TestMirroring)

[7241456]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241456/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241456/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ec59e119ecd246c5a048c36b9445dfa3

    Command failed on smithi142 with status 1: 'sudo yum upgrade -y linux-
    firmware'

[7241469]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:16:39
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241469/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241469/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0b331e7749b2459882c1a01a63f1d656

    more than one file system available

[7241478]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:14:36
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241478/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241478/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c047d6c9e4a54769b9b48232bb759b97

    more than one file system available

[7241482]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   02:01:29
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241482/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241482/

    "2023-04-15T01:00:03.632473+0000 mon.a (mon.0) 1673 : cluster [WRN] Health
    check failed: Degraded data redundancy: 1/46 objects degraded (2.174%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[7241500]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   01:13:59
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241500/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241500/
sentry: https://sentry.ceph.com/organizations/ceph/?query=580b985291fa45d1aa42369f2c5d7206

    more than one file system available

[7241508]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:16:45
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241508/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241508/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e7075bb73aad433d8faeadb2e1c00878

    more than one file system available

[7241519]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:14:40
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241519/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241519/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f431486824244538946209e1a3f7ed8e

    more than one file system available

[7241540]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   01:14:52
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241540/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241540/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cdc3124b881f40bc9c73c72d0e803fe7

    more than one file system available

[7241549]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:16:52
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241549/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241549/
sentry: https://sentry.ceph.com/organizations/ceph/?query=041f975be3494c9ab82317a5211dff4b

    more than one file system available

[7241563]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:17:12
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241563/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241563/
sentry: https://sentry.ceph.com/organizations/ceph/?query=56e6c2cff80c4d58a24e6f0996de9be4

    more than one file system available

[7241572]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:14:41
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241572/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241572/
sentry: https://sentry.ceph.com/organizations/ceph/?query=215a138a44494a37bd5cb433dca3a11a

    more than one file system available



Dead
=================================================================
[7241413]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:08:37
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241413/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241413/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5fe527beb41a4fb48d1df43666e0d45d

    {'smithi023.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}, 'smithi083.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}}

[7241419]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:04:16
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241419/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241419/
sentry: https://sentry.ceph.com/organizations/ceph/?query=464615f57af447d2b92d30ef6708880e

    {'smithi093.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}, 'smithi031.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []}}

[7241420]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:41:07
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241420/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241420/
sentry: https://sentry.ceph.com/organizations/ceph/?query=041d1a86d7b64a7795dd901c3e34eec5

    {'smithi102.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/25530bb773028965
    8577cd6a7aed7adb251d5fb40a42ebce2cb1f6d1cf1b1f87-filelists.xml.gz - Cannot
    download, all mirrors were already tried without success", 'rc': 1,
    'results': []}}

[7241421]  fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:04:10
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241421/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241421/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a25a8a451aeb479c88e4923d780d19ef

    {'smithi145.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.67)", 'rc': 1, 'results': []}, 'smithi151.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}}

[7241424]  fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
-----------------------------------------------------------------
time:   00:04:12
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241424/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241424/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c25da0187ec54bdfb8d72ca5735d5c6f

    {'smithi150.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []}}

[7241428]  fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:04:18
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241428/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241428/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a88c0d9869b0496fbc4e047875f0591a

    {'smithi139.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.67)", 'rc': 1, 'results': []}, 'smithi019.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}}

[7241431]  fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:04:15
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241431/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241431/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4062faa40bd471e959e5b64d7d574e5

    {'smithi144.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.67)", 'rc': 1, 'results': []}, 'smithi135.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}}

[7241434]  fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/fstop}
-----------------------------------------------------------------
time:   00:04:09
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241434/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241434/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5639ab0b2e0b4ffea86100df5c892101

    {'smithi100.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []}}

[7241435]  fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
-----------------------------------------------------------------
time:   00:04:18
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241435/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241435/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a1c8089ba24e471b8ddb61002d25f249

    {'smithi179.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []},
    'smithi037.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}}

[7241440]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}}
-----------------------------------------------------------------
time:   00:08:49
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241440/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241440/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e97bee00d3e5444ca553dbd3d82713ae

    {'smithi019.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []},
    'smithi177.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}, 'smithi033.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}}

[7241441]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:04:22
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241441/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241441/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b079881d7a0a40ebacd88171e4027fdc

    {'smithi143.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []},
    'smithi130.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}}

[7241443]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:04:13
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241443/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241443/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3419f298af354b168951051447763f1d

    {'smithi135.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}, 'smithi144.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}}

[7241446]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
-----------------------------------------------------------------
time:   00:04:21
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241446/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241446/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aed3c980a7e04bcd882c07780445b498

    {'smithi142.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}, 'smithi131.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.67)", 'rc': 1, 'results': []}}

[7241447]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
-----------------------------------------------------------------
time:   00:40:50
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241447/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241447/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bc0052dcc9ba41ddadaf24f10faf23c3

    {'smithi188.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s):
    repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-
    comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already
    tried without success", 'rc': 1, 'results': []}}

[7241448]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:04:09
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241448/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241448/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7415e038331b4f629b65cb72069b9852

    {'smithi111.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}, 'smithi097.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}}

[7241452]  fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:04:18
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241452/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241452/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a3ffe3fdb5084a41bc992c00d8d185f8

    {'smithi154.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}, 'smithi019.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []}}

[7241453]  fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:04:14
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241453/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241453/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d361d46f03a54da19c0222ae00d55910

    {'smithi135.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}, 'smithi144.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}}

[7241457]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:04:15
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241457/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241457/
sentry: https://sentry.ceph.com/organizations/ceph/?query=18a11350261042519ccae51296be8092

    {'smithi097.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []},
    'smithi111.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}}

[7241458]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
-----------------------------------------------------------------
time:   00:04:21
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241458/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241458/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c2c4b68bdfb4449a928f0776eafd6d30

    {'smithi173.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}, 'smithi050.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.67)", 'rc': 1, 'results': []}}

[7241459]  fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   00:08:39
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241459/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241459/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0147a59d11594c4eb376acf41b484d75

    {'smithi046.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []},
    'smithi181.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}}

[7241463]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
-----------------------------------------------------------------
time:   00:04:20
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241463/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241463/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9737626b150f40a5a4f103a01a2ce5f1

    {'smithi100.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.67)", 'rc': 1, 'results': []}, 'smithi026.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []}}

[7241464]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:08:43
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241464/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241464/
sentry: https://sentry.ceph.com/organizations/ceph/?query=960a44c453ba4ff396317981fd4f6e3e

    {'smithi019.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}, 'smithi154.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}}

[7241465]  fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:04:12
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241465/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241465/
sentry: https://sentry.ceph.com/organizations/ceph/?query=111bd9cc60f24a1084c985b4e2f9390e

    {'smithi135.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}, 'smithi144.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}}

[7241466]  fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:08:37
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241466/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241466/
sentry: https://sentry.ceph.com/organizations/ceph/?query=59cb19bb875e45fe9702ea2235bbec1b

    {'smithi060.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}, 'smithi156.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}}

[7241468]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
-----------------------------------------------------------------
time:   00:40:53
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241468/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241468/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fc995c907137414f8177c4d896851514

    {'smithi182.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s):
    repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-
    comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already
    tried without success", 'rc': 1, 'results': []}}

[7241476]  fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:04:13
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241476/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241476/
sentry: https://sentry.ceph.com/organizations/ceph/?query=36a176978cca4b6dbf17eff3de3e5325

    {'smithi142.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []},
    'smithi100.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}}

[7241479]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
-----------------------------------------------------------------
time:   00:04:16
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241479/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241479/
sentry: https://sentry.ceph.com/organizations/ceph/?query=294b1e448146496ba021c375a03b870f

    {'smithi119.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}, 'smithi043.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}}

[7241480]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}}
-----------------------------------------------------------------
time:   00:08:43
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241480/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241480/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3b39b4f905954d5185686b59d5eacfa1

    {'smithi032.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []},
    'smithi170.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}, 'smithi161.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}}

[7241488]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:04:20
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241488/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241488/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b983c5a4c0524495834caaaf2404381d

    {'smithi203.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []},
    'smithi047.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}}

[7241490]  fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:04:10
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241490/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241490/
sentry: https://sentry.ceph.com/organizations/ceph/?query=67a47a4d32c148c2b983fb2989fd501c

    {'smithi100.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}, 'smithi072.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}}

[7241493]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:04:23
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241493/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241493/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eeeb439e32854570aeb2ee2e17be1374

    {'smithi026.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []},
    'smithi006.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}}

[7241498]  fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
-----------------------------------------------------------------
time:   00:04:16
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241498/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241498/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5fb48d58a86a452f8b834ac8462291f0

    {'smithi158.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}, 'smithi044.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []}}

[7241499]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
-----------------------------------------------------------------
time:   00:04:14
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241499/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241499/
sentry: https://sentry.ceph.com/organizations/ceph/?query=544a38ce9ab043dd86d321f9f726b769

    {'smithi144.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}, 'smithi135.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}}

[7241503]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241503/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241503/

    hit max job timeout

[7241505]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:08:45
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241505/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241505/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e7a3c17b39364e61a3c084d1886751f6

    {'smithi167.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}, 'smithi188.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}}

[7241509]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
-----------------------------------------------------------------
time:   00:38:40
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241509/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241509/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4c6958a06ee34f818691f0f68058d105

    {'smithi049.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools':
    Yum repo downloading error: Downloading error(s): repodata/16bdfd9c9155376d
    dc8f552e5bcb7a6deb94739e35f439ba1a5db715bac5ddd9-primary.xml.gz - Cannot
    download, all mirrors were already tried without success; repodata/25530bb7
    730289658577cd6a7aed7adb251d5fb40a42ebce2cb1f6d1cf1b1f87-filelists.xml.gz -
    Cannot download, all mirrors were already tried without success;
    repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-
    comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already
    tried without success", 'rc': 1, 'results': []}}

[7241510]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}}
-----------------------------------------------------------------
time:   00:04:34
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241510/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241510/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a5789dc7713403fa850835381d99287

    {'smithi171.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []},
    'smithi044.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}, 'smithi158.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}}

[7241511]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
-----------------------------------------------------------------
time:   00:04:29
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241511/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241511/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2288ab5fd8054e3dbc2ce7beb39cd091

    {'smithi135.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []},
    'smithi144.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}}

[7241515]  fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:04:18
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241515/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241515/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0348653512574984a5a2f0a05c49ce52

    {'smithi110.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.67)", 'rc': 1, 'results': []}, 'smithi019.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}}

[7241517]  fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:04:16
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241517/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241517/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7ad7414cfe5a4beab62912c1eafaafca

    {'smithi097.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []},
    'smithi111.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}}

[7241522]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
-----------------------------------------------------------------
time:   00:08:47
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241522/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241522/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ae45e19896a14d9cbff9f337e7ee912c

    {'smithi057.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []},
    'smithi178.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []},
    'smithi162.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []}}

[7241524]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:04:19
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241524/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241524/
sentry: https://sentry.ceph.com/organizations/ceph/?query=072221494f8a4ea98a93c9fba2844228

    {'smithi093.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}, 'smithi031.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []}}

[7241528]  fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
-----------------------------------------------------------------
time:   00:04:20
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241528/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241528/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fe23c56960244fc980422e3eac9b8776

    {'smithi150.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.21)", 'rc': 1, 'results': []}, 'smithi139.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}}

[7241529]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
-----------------------------------------------------------------
time:   00:04:17
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241529/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241529/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b675a924a81341068cb6e6ecd4a23e26

    {'smithi182.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []},
    'smithi173.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []}}

[7241530]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:04:15
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241530/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241530/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0669bed52d244d4199e1121bf91ab6e4

    {'smithi084.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}, 'smithi122.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}}

[7241546]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
-----------------------------------------------------------------
time:   00:04:14
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241546/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241546/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1028b48680044c8d93b8f878df8d4b31

    {'smithi196.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}, 'smithi081.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []}}

[7241551]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
-----------------------------------------------------------------
time:   00:04:23
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241551/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241551/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e2b0e9d64d1c40588c1b2904a8a22985

    {'smithi179.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}, 'smithi037.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}, 'smithi059.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []}}

[7241552]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:08:36
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241552/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241552/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b84bd1ffa5b349df98fdb7338c1e9ec5

    {'smithi081.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []},
    'smithi196.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.67)", 'rc': 1, 'results': []}}

[7241559]  fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
-----------------------------------------------------------------
time:   00:04:21
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241559/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241559/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c61bb2dc2a004dee85663779547acdd4

    {'smithi153.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}, 'smithi045.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}}

[7241560]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client}
-----------------------------------------------------------------
time:   00:04:15
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241560/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241560/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8ebf96d5d3c34a7f8bed5baa2c8b4a5e

    {'smithi159.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []},
    'smithi132.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []}}

[7241561]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:04:15
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241561/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241561/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b3c3aa08275b459896bc53bd0bdb7842

    {'smithi122.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.67)", 'rc': 1, 'results': []}, 'smithi084.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    34.221.3.152)", 'rc': 1, 'results': []}}

[7241564]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
-----------------------------------------------------------------
time:   00:04:22
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241564/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241564/
sentry: https://sentry.ceph.com/organizations/ceph/?query=02e8c1aac9ed480a94e64358fd9db468

    {'smithi158.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    8.43.85.73)", 'rc': 1, 'results': []}, 'smithi044.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []}}

[7241567]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
-----------------------------------------------------------------
time:   00:08:49
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241567/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241567/
sentry: https://sentry.ceph.com/organizations/ceph/?query=644b2fea096c4e0f93733577a86d7e85

    {'smithi029.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.142)", 'rc': 1, 'results': []},
    'smithi081.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []},
    'smithi193.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    152.19.134.198)", 'rc': 1, 'results': []}}

[7241573]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:04:17
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241573/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241573/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6bd54f51d51a42bb8406cf89ada42b7a

    {'smithi187.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}, 'smithi078.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    67.219.144.68)", 'rc': 1, 'results': []}}

[7241576]  fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:04:23
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241576/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241576/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9fbc748300ba4544990fb1bf661680a3

    {'smithi139.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}, 'smithi063.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'present',
    'update_cache': False, 'update_only': False, 'validate_certs': True}},
    'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    209.132.190.2)", 'rc': 1, 'results': []}}

[7241578]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
-----------------------------------------------------------------
time:   00:04:13
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241578/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241578/
sentry: https://sentry.ceph.com/organizations/ceph/?query=19a412a0ff234a5d99a5450d27848455

    {'smithi120.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    140.211.169.196)", 'rc': 1, 'results': []},
    'smithi093.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'],
    'releasever': None, 'security': False, 'skip_broken': False, 'state':
    'present', 'update_cache': False, 'update_only': False, 'validate_certs':
    True}}, 'msg': "Failed to download metadata for repo 'epel': Cannot prepare
    internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org
    /metalink?repo=epel-8.4&arch=x86_64&infra=$infra&content=$contentdir (IP:
    38.145.60.20)", 'rc': 1, 'results': []}}



Running
=================================================================
[7241568] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241568/



Pass
=================================================================
[7241408] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:56:28
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241408/

[7241409] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}}
time:   00:16:10
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241409/

[7241411] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   01:13:41
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241411/

[7241412] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:26:44
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241412/

[7241414] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:55:15
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241414/

[7241415] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   01:21:13
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241415/

[7241416] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   01:27:43
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241416/

[7241417] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind}
time:   01:54:00
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241417/

[7241422] fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   01:09:12
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241422/

[7241423] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest}
time:   02:02:49
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241423/

[7241425] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   01:12:16
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241425/

[7241427] fs/mirror-ha/{begin cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit}
time:   02:38:49
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241427/

[7241429] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:48:30
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241429/

[7241430] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:30:03
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241430/

[7241432] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:40:04
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241432/

[7241433] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:49:18
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241433/

[7241436] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   01:55:18
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241436/

[7241438] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client}
time:   01:26:48
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241438/

[7241439] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   01:14:35
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241439/

[7241442] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:15:27
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241442/

[7241444] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:14:48
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241444/

[7241445] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:48:31
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241445/

[7241449] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:48:21
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241449/

[7241450] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}}
time:   01:32:32
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241450/

[7241451] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   01:18:30
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241451/

[7241454] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:14:09
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241454/

[7241455] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
time:   00:17:32
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241455/

[7241460] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   02:35:40
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241460/

[7241461] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:11:55
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241461/

[7241462] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
time:   01:49:52
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241462/

[7241467] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:53:09
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241467/

[7241470] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}}
time:   01:14:07
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241470/

[7241471] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:13:37
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241471/

[7241472] fs/upgrade/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04}
time:   01:13:40
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241472/

[7241473] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   02:28:52
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241473/

[7241474] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   01:25:18
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241474/

[7241475] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:12:08
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241475/

[7241477] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   01:22:23
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241477/

[7241481] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   02:03:03
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241481/

[7241483] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   01:39:07
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241483/

[7241484] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:38:45
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241484/

[7241485] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
time:   00:27:28
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241485/

[7241486] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:36:09
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241486/

[7241487] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:18:39
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241487/

[7241489] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}}
time:   01:35:24
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241489/

[7241491] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   02:02:15
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241491/

[7241492] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   00:32:56
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241492/

[7241494] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   01:13:35
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241494/

[7241495] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   01:25:19
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241495/

[7241496] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   01:11:31
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241496/

[7241497] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:46:47
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241497/

[7241501] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   01:22:29
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241501/

[7241502] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
time:   00:30:48
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241502/

[7241504] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:21:27
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241504/

[7241506] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:22:20
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241506/

[7241507] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:18:24
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241507/

[7241512] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   01:31:21
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241512/

[7241513] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:35
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241513/

[7241514] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:15:56
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241514/

[7241516] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:48:41
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241516/

[7241518] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   01:29:19
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241518/

[7241520] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:15:52
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241520/

[7241521] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:50:59
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241521/

[7241523] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
time:   02:48:57
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241523/

[7241525] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:17:02
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241525/

[7241526] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:51:06
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241526/

[7241527] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:11:19
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241527/

[7241531] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   01:52:24
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241531/

[7241532] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:32:33
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241532/

[7241533] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   01:12:48
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241533/

[7241534] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
time:   01:16:04
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241534/

[7241535] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:57:07
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241535/

[7241536] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   01:20:38
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241536/

[7241537] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   01:11:49
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241537/

[7241538] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:14:01
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241538/

[7241539] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:54:10
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241539/

[7241541] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
time:   01:11:27
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241541/

[7241542] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   01:36:43
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241542/

[7241543] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:14:15
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241543/

[7241544] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}}
time:   00:19:57
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241544/

[7241545] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:55:59
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241545/

[7241547] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
time:   01:27:27
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241547/

[7241548] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   01:46:34
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241548/

[7241550] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:16:30
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241550/

[7241553] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep}
time:   01:08:20
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241553/

[7241554] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:28
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241554/

[7241555] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:11:43
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241555/

[7241556] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:24:36
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241556/

[7241557] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:08
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241557/

[7241558] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:45:03
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241558/

[7241562] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   02:11:57
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241562/

[7241565] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   01:16:42
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241565/

[7241566] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   01:28:52
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241566/

[7241569] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:32:41
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241569/

[7241570] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:52:23
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241570/

[7241571] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:14:43
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241571/

[7241574] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:16:04
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241574/

[7241575] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   01:25:48
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241575/

[7241577] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:14:05
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241577/

[7241579] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   01:58:07
info:   http://pulpito.ceph.com/yuriw-2023-04-14_15:35:25-fs-wip-yuri4-testing-2023-04-11-1247-pacific-distro-default-smithi/7241579/