2022-06-30T19:06:34.907 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3
2022-06-30T19:06:34.909 INFO:teuthology.results:seed: '7707'
2022-06-30T22:06:39.467 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3
2022-06-30T22:06:39.509 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-07-01T00:22:52.617 INFO:teuthology.results:Tests finished! gathering results...
2022-07-01T00:23:04.158 INFO:scrape:Found 96 jobs
2022-07-01T00:23:14.192 INFO:scrape:Found 8 distinct failure reasons
2022-07-01T00:23:14.194 INFO:scrape:Failure: {'smithi032.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}}
2022-07-01T00:23:14.194 INFO:scrape:30 jobs
2022-07-01T00:23:14.195 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'session_timeout}', 'standby-replay']
2022-07-01T00:23:14.196 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2022-07-01T00:23:14.197 INFO:scrape:
2022-07-01T00:23:14.197 INFO:scrape:Failure: Command failed on smithi026 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"
2022-07-01T00:23:14.198 INFO:scrape:9 jobs: ['6907631', '6907633', '6907641', '6907643', '6907647', '6907605', '6907653', '6907580', '6907616']
2022-07-01T00:23:14.199 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{frag', 'session_timeout}', 'standby-replay']
2022-07-01T00:23:14.199 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2022-07-01T00:23:14.200 INFO:scrape:
2022-07-01T00:23:14.200 INFO:scrape:Failure: Command failed on smithi018 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b2ff29d7b7232005ab121a1061d52f563e83dab6 shell --fsid b3ce855a-f8b6-11ec-842c-001a4aab830c -- ceph daemon mds.a perf dump'
2022-07-01T00:23:14.201 INFO:scrape:2 jobs: ['6907632', '6907615']
2022-07-01T00:23:14.201 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout}', 'standby-replay', 'wsync/yes}']
2022-07-01T00:23:14.202 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/5', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2022-07-01T00:23:14.202 INFO:scrape:
2022-07-01T00:23:14.202 INFO:scrape:Failure: '2022-06-30T20:17:01.954800+00:00 smithi105 kernel: kernel BUG at mm/filemap.c:1559!
' in syslog
2022-07-01T00:23:14.203 INFO:scrape:4 jobs: ['6907585', '6907614', '6907622', '6907598']
2022-07-01T00:23:14.203 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout}', 'standby-replay', 'wsync/yes}']
2022-07-01T00:23:14.204 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/3', 'ranks/5', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}']
2022-07-01T00:23:14.204 INFO:scrape:
2022-07-01T00:23:14.205 INFO:scrape:Failure: Command failed on smithi195 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib'
2022-07-01T00:23:14.205 INFO:scrape:1 jobs: ['6907612']
2022-07-01T00:23:14.206 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/3', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}']
2022-07-01T00:23:14.206 INFO:scrape:
2022-07-01T00:23:14.207 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi055.front.sepia.ceph.com: ['type=AVC msg=audit(1656616649.796:193): avc:  denied  { node_bind } for  pid=1406 comm="ping" saddr=172.21.15.55 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1']
2022-07-01T00:23:14.208 INFO:scrape:1 jobs: ['6907568']
2022-07-01T00:23:14.209 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}']
2022-07-01T00:23:14.209 INFO:scrape:
2022-07-01T00:23:14.210 INFO:scrape:Failure: {'smithi093.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'dest': '/tmp/katello-ca-consumer-latest.noarch.rpm', 'elapsed': 0, 'gid': 0, 'group': 'root', 'invocation': {'module_args': {'attributes': None, 'backup': False, 'checksum': '', 'client_cert': None, 'client_key': None, 'dest': '/tmp/katello-ca-consumer-latest.noarch.rpm', 'force': False, 'force_basic_auth': False, 'group': None, 'headers': None, 'http_agent': 'ansible-httpget', 'mode': None, 'owner': None, 'selevel': None, 'serole': None, 'setype': None, 'seuser': None, 'sha256sum': '', 'timeout': 10, 'tmp_dest': None, 'unsafe_writes': False, 'url': 'http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-latest.noarch.rpm', 'url_password': None, 'url_username': None, 'use_proxy': True, 'validate_certs': True}}, 'mode': '0644', 'msg': 'Request failed: <urlopen error [Errno 111] Connection refused>', 'owner': 'root', 'secontext': 'unconfined_u:object_r:user_home_t:s0', 'size': 7419, 'state': 'file', 'uid': 0, 'url': 'http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-latest.noarch.rpm'}, 'smithi117.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'dest': '/tmp/katello-ca-consumer-latest.noarch.rpm', 'elapsed': 0, 'gid': 0, 'group': 'root', 'invocation': {'module_args': {'attributes': None, 'backup': False, 'checksum': '', 'client_cert': None, 'client_key': None, 'dest': '/tmp/katello-ca-consumer-latest.noarch.rpm', 'force': False, 'force_basic_auth': False, 'group': None, 'headers': None, 'http_agent': 'ansible-httpget', 'mode': None, 'owner': None, 'selevel': None, 'serole': None, 'setype': None, 'seuser': None, 'sha256sum': '', 'timeout': 10, 'tmp_dest': None, 'unsafe_writes': False, 'url': 'http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-latest.noarch.rpm', 'url_password': None, 'url_username': None, 'use_proxy': True, 'validate_certs': True}}, 'mode': '0644', 'msg': 'Request failed: <urlopen error [Errno 111] Connection refused>', 'owner': 'root', 'secontext': 'unconfined_u:object_r:user_home_t:s0', 'size': 7419, 'state': 'file', 'uid': 0, 'url': 'http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-latest.noarch.rpm'}}
2022-07-01T00:23:14.211 INFO:scrape:1 jobs: ['6907617']
2022-07-01T00:23:14.211 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}']
2022-07-01T00:23:14.212 INFO:scrape:
2022-07-01T00:23:14.213 INFO:scrape:Failure: {'smithi064.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi090.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['dnf-utils', 'git-all', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'podman', 'redhat-lsb', 'firewalld', 'wget', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'make', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu-kvm-block-rbd', 'libacl-devel', 'autoconf', 'gdb', 'iozone', 'lvm2'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", 'rc': 1, 'results': []}, 'smithi047.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['ceph', 'ceph-base', 'ceph-selinux', 'ceph-common', 'ceph-debuginfo', 'ceph-release', 'libcephfs1', 'ceph-radosgw', 'python-ceph', 'python-rados', 'python-rbd', 'python-cephfs', 'librbd1', 'librados2', 'mod_fastcgi'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'absent', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", 'rc': 1, 'results': []}}
2022-07-01T00:23:14.214 INFO:scrape:1 jobs: ['6907618']
2022-07-01T00:23:14.214 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-workunit/db/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}']
2022-07-01T00:23:14.215 INFO:scrape:
2022-07-01T00:23:14.222 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/
failed:  19
dead:    30
running: 0
waiting: 0
queued:  0
passed:  47


Fail
=================================================================
[6907568]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:54:02
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907568/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907568/

    SELinux denials found on ubuntu@smithi055.front.sepia.ceph.com: ['type=AVC
    msg=audit(1656616649.796:193): avc:  denied  { node_bind } for  pid=1406
    comm="ping" saddr=172.21.15.55 scontext=system_u:system_r:ping_t:s0
    tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1']

[6907580]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:53:51
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907580/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907580/

    Command failed on smithi162 with status 123: "find
    /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0
    --no-run-if-empty -- gzip --"

[6907585]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:31:07
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907585/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907585/

    '2022-06-30T20:17:01.954800+00:00 smithi105 kernel: kernel BUG at
    mm/filemap.c:1559! ' in syslog

[6907592]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:09:06
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907592/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907592/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dad869e7232d45f5821a69c0eda624c9

    {'smithi017.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907598]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:48:35
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907598/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907598/

    '2022-06-30T20:29:15.144359+00:00 smithi050 kernel: kernel BUG at
    mm/filemap.c:1559! ' in syslog

[6907605]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:57:52
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907605/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907605/

    Command failed on smithi080 with status 123: "find
    /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0
    --no-run-if-empty -- gzip --"

[6907607]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:43
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907607/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907607/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5a5636d7f4c546fd9c04720e5c65f1c2

    {'smithi040.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907612]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:08
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907612/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907612/
sentry: https://sentry.ceph.com/organizations/ceph/?query=522ade91297343edaa572fd5c0103260

    Command failed on smithi195 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel
    librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-
    fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-
    contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager
    iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-
    devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel
    xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-
    devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-
    devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel
    libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-
    server postgresql-contrib'

[6907614]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:49:40
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907614/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907614/

    '2022-06-30T20:48:19.621765+00:00 smithi046 kernel: kernel BUG at
    mm/filemap.c:1559! ' in syslog

[6907615]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:25:23
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907615/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907615/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d4799c74911e455abf1200bcbed0e3dc

    Command failed on smithi139 with status 22: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:b2ff29d7b7232005ab121a1061d52f563e83dab6 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    a952e6e8-f8b1-11ec-842c-001a4aab830c -- ceph orch daemon add osd
    smithi139:vg_nvme/lv_4'

[6907616]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:49:58
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907616/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907616/

    Command failed on smithi003 with status 1: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph fs dump --format=json'

[6907622]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:48:09
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907622/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907622/

    '2022-06-30T20:56:08.388413+00:00 smithi040 kernel: kernel BUG at
    mm/filemap.c:1559! ' in syslog

[6907631]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:51:26
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907631/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907631/

    Command failed on smithi026 with status 123: "find
    /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0
    --no-run-if-empty -- gzip --"

[6907632]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:43:12
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907632/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907632/

    Command failed on smithi018 with status 127: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:b2ff29d7b7232005ab121a1061d52f563e83dab6 shell --fsid
    b3ce855a-f8b6-11ec-842c-001a4aab830c -- ceph daemon mds.a perf dump'

[6907633]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:56:44
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907633/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907633/

    Command failed on smithi062 with status 1: "sudo
    TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status
    --fs=cephfs --path=/'"

[6907641]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   01:34:38
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907641/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907641/

    Command failed on smithi045 with status 1: "sudo
    TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status
    --fs=cephfs --path=/'"

[6907643]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:45:58
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907643/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907643/

    Command failed on smithi162 with status 123: "find
    /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0
    --no-run-if-empty -- gzip --"

[6907647]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:55:45
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907647/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907647/

    Command failed on smithi026 with status 123: "find
    /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0
    --no-run-if-empty -- gzip --"

[6907653]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   02:12:54
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907653/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907653/

    Command failed on smithi080 with status 123: "find
    /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0
    --no-run-if-empty -- gzip --"



Dead
=================================================================
[6907566]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:56
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907566/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907566/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a3611bf421cf455981bcc4da41b68265

    {'smithi018.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907567]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:54
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907567/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907567/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dbfaa91571b24ca7a67061e1ee528f50

    {'smithi150.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907569]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:09:03
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907569/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907569/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fcf04cb3e39a404882b9b8f621102e75

    {'smithi017.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907573]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:07:41
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907573/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907573/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b6aab2a5724e409d921622733dedfd19

    {'smithi032.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907574]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:57
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907574/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907574/
sentry: https://sentry.ceph.com/organizations/ceph/?query=34ea74a607ce4199b91df28957fff41f

    {'smithi086.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907575]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:57
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907575/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907575/
sentry: https://sentry.ceph.com/organizations/ceph/?query=beeb96d8dcd442ada98429100933dd9c

    {'smithi145.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}, 'smithi114.front.sepia.ceph.com': {'attempts':
    12, 'censored': "the output has been hidden due to the fact that 'no_log:
    true' was specified for this result", 'changed': True}}

[6907576]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:09:02
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907576/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907576/
sentry: https://sentry.ceph.com/organizations/ceph/?query=81b2f3d33f614a6cb11d325173e0f5ff

    {'smithi026.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907577]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:49
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907577/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907577/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3bd063aa3c724574a40bf9f2cd12a33b

    {'smithi120.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907582]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:50
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907582/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907582/
sentry: https://sentry.ceph.com/organizations/ceph/?query=114db909f8174cd3a2d140878724727c

    {'smithi080.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907583]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:53
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907583/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907583/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4229de24776463fb99b159b0440b221

    {'smithi043.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907584]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:58
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907584/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907584/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6fc4e6894fc0484c8d754580e0d40159

    {'smithi002.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907586]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:07:22
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907586/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907586/
sentry: https://sentry.ceph.com/organizations/ceph/?query=81aa7d3d2d384f4bb684b7fcccc027ab

    {'smithi046.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907588]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:07:25
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907588/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907588/
sentry: https://sentry.ceph.com/organizations/ceph/?query=89a550577e124cb98ae2c96a5d93cc26

    {'smithi032.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907590]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:09:01
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907590/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907590/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e416f39f206f49489eee05fdcb52b0b5

    {'smithi102.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907591]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:09:00
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907591/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907591/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a561ca5fedd3470dafa100373b819205

    {'smithi079.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907593]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:09:05
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907593/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907593/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dbcff64c96cb4e3a9c3eafca53b98aa3

    {'smithi019.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907594]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:07:41
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907594/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907594/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ddde0b4852ed48b7a917521d6806df25

    {'smithi033.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907597]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:07:42
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907597/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907597/
sentry: https://sentry.ceph.com/organizations/ceph/?query=175785c8b15a40fb8c7e0b1f10da1f2f

    {'smithi145.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907599]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:09:02
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907599/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907599/
sentry: https://sentry.ceph.com/organizations/ceph/?query=42d0517bc69e4839a1c5064da0d6a5c3

    {'smithi026.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907600]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:55
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907600/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907600/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9b33aa82054c4828ae7c731fde6315d0

    {'smithi074.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907601]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:59
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907601/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907601/
sentry: https://sentry.ceph.com/organizations/ceph/?query=480fd33b7b604f0686308251b1ec45e4

    {'smithi038.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}, 'smithi003.front.sepia.ceph.com': {'attempts':
    12, 'censored': "the output has been hidden due to the fact that 'no_log:
    true' was specified for this result", 'changed': True}}

[6907603]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:07:14
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907603/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907603/
sentry: https://sentry.ceph.com/organizations/ceph/?query=945764a6e9e64e1b9968ba3673bcd734

    {'smithi046.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907604]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:07:19
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907604/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907604/
sentry: https://sentry.ceph.com/organizations/ceph/?query=74cd2b897fe44748956b30c93efba5dc

    {'smithi032.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907606]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:45
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907606/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907606/
sentry: https://sentry.ceph.com/organizations/ceph/?query=21868344aea8460f851a0b773bec1667

    {'smithi043.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907608]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:35
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907608/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907608/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b0d5dcd4c87a49568c1e253fd04fc6df

    {'smithi102.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907609]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:08:40
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907609/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907609/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4bcd9baaf1ab4052987670bdf4a532e9

    {'smithi017.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907611]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:06:51
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907611/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907611/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d061af95f79a41ffbf6c2123c2f621ba

    {'smithi145.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907613]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:07:01
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907613/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907613/
sentry: https://sentry.ceph.com/organizations/ceph/?query=378ff28542dc4ca49eea82639f8ddd05

    {'smithi032.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6907617]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:10:51
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907617/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907617/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bac74b74cc014e66bee2d5b7cbd92acd

    {'smithi093.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'dest': '/tmp/katello-ca-consumer-latest.noarch.rpm', 'elapsed': 0,
    'gid': 0, 'group': 'root', 'invocation': {'module_args': {'attributes':
    None, 'backup': False, 'checksum': '', 'client_cert': None, 'client_key':
    None, 'dest': '/tmp/katello-ca-consumer-latest.noarch.rpm', 'force': False,
    'force_basic_auth': False, 'group': None, 'headers': None, 'http_agent':
    'ansible-httpget', 'mode': None, 'owner': None, 'selevel': None, 'serole':
    None, 'setype': None, 'seuser': None, 'sha256sum': '', 'timeout': 10,
    'tmp_dest': None, 'unsafe_writes': False, 'url':
    'http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-
    latest.noarch.rpm', 'url_password': None, 'url_username': None,
    'use_proxy': True, 'validate_certs': True}}, 'mode': '0644', 'msg':
    'Request failed: <urlopen error [Errno 111] Connection refused>', 'owner':
    'root', 'secontext': 'unconfined_u:object_r:user_home_t:s0', 'size': 7419,
    'state': 'file', 'uid': 0, 'url':
    'http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-
    latest.noarch.rpm'}, 'smithi117.front.sepia.ceph.com': {'_ansible_no_log':
    False, 'changed': False, 'dest': '/tmp/katello-ca-consumer-
    latest.noarch.rpm', 'elapsed': 0, 'gid': 0, 'group': 'root', 'invocation':
    {'module_args': {'attributes': None, 'backup': False, 'checksum': '',
    'client_cert': None, 'client_key': None, 'dest': '/tmp/katello-ca-consumer-
    latest.noarch.rpm', 'force': False, 'force_basic_auth': False, 'group':
    None, 'headers': None, 'http_agent': 'ansible-httpget', 'mode': None,
    'owner': None, 'selevel': None, 'serole': None, 'setype': None, 'seuser':
    None, 'sha256sum': '', 'timeout': 10, 'tmp_dest': None, 'unsafe_writes':
    False, 'url': 'http://satellite.front.sepia.ceph.com/pub/katello-ca-
    consumer-latest.noarch.rpm', 'url_password': None, 'url_username': None,
    'use_proxy': True, 'validate_certs': True}}, 'mode': '0644', 'msg':
    'Request failed: <urlopen error [Errno 111] Connection refused>', 'owner':
    'root', 'secontext': 'unconfined_u:object_r:user_home_t:s0', 'size': 7419,
    'state': 'file', 'uid': 0, 'url':
    'http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-
    latest.noarch.rpm'}}

[6907618]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
-----------------------------------------------------------------
time:   00:05:36
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907618/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907618/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aeee2e75212142a9a2149981008045f9

    {'smithi064.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}, 'smithi090.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args':
    {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False,
    'bugfix': False, 'conf_file': None, 'disable_excludes': None,
    'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [],
    'download_dir': None, 'download_only': False, 'enable_plugin': [],
    'enablerepo': [], 'exclude': [], 'install_repoquery': True,
    'install_weak_deps': True, 'installroot': '/', 'list': None,
    'lock_timeout': 30, 'name': ['dnf-utils', 'git-all', 'sysstat', 'libedit',
    'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs',
    'openssl', 'libuuid', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump',
    'blktrace', 'usbredir', 'podman', 'redhat-lsb', 'firewalld', 'wget',
    'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git',
    'make', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img',
    'qemu-kvm-core', 'qemu-kvm-block-rbd', 'libacl-devel', 'autoconf', 'gdb',
    'iozone', 'lvm2'], 'releasever': None, 'security': False, 'skip_broken':
    False, 'state': 'present', 'update_cache': False, 'update_only': False,
    'validate_certs': True}}, 'msg': "Failed to download metadata for repo
    'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot
    download repodata/repomd.xml: All mirrors were tried", 'rc': 1, 'results':
    []}, 'smithi047.front.sepia.ceph.com': {'_ansible_no_log': False,
    'changed': False, 'invocation': {'module_args': {'allow_downgrade': False,
    'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file':
    None, 'disable_excludes': None, 'disable_gpg_check': False,
    'disable_plugin': [], 'disablerepo': [], 'download_dir': None,
    'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude':
    [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot':
    '/', 'list': None, 'lock_timeout': 30, 'name': ['ceph', 'ceph-base', 'ceph-
    selinux', 'ceph-common', 'ceph-debuginfo', 'ceph-release', 'libcephfs1',
    'ceph-radosgw', 'python-ceph', 'python-rados', 'python-rbd', 'python-
    cephfs', 'librbd1', 'librados2', 'mod_fastcgi'], 'releasever': None,
    'security': False, 'skip_broken': False, 'state': 'absent', 'update_cache':
    False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to
    download metadata for repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download
    repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried",
    'rc': 1, 'results': []}}



Pass
=================================================================
[6907570] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:03:20
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907570/

[6907571] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:57:46
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907571/

[6907572] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:07:12
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907572/

[6907578] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:44:55
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907578/

[6907579] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:44:36
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907579/

[6907581] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:27:20
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907581/

[6907587] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:55:59
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907587/

[6907589] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:00:38
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907589/

[6907595] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:44:44
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907595/

[6907596] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:51:09
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907596/

[6907602] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:59:27
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907602/

[6907610] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:14:33
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907610/

[6907619] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:58:36
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907619/

[6907620] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:12:18
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907620/

[6907621] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:08:59
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907621/

[6907623] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:50:31
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907623/

[6907624] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:54:47
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907624/

[6907625] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:34:29
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907625/

[6907626] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:23:36
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907626/

[6907627] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:44:23
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907627/

[6907628] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:01:11
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907628/

[6907629] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:37:52
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907629/

[6907630] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:46:29
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907630/

[6907634] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:02:10
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907634/

[6907635] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:55:49
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907635/

[6907636] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:08:49
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907636/

[6907637] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:36:50
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907637/

[6907638] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:48:33
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907638/

[6907639] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:52:28
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907639/

[6907640] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:52:11
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907640/

[6907642] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:46:12
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907642/

[6907644] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:29:47
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907644/

[6907645] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:29:32
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907645/

[6907646] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:44:59
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907646/

[6907648] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:53:12
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907648/

[6907649] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:37:14
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907649/

[6907650] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:04:01
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907650/

[6907651] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:56:12
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907651/

[6907652] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:12:55
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907652/

[6907654] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:51:05
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907654/

[6907655] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:52:45
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907655/

[6907656] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:20:00
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907656/

[6907657] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:40:41
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907657/

[6907658] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:01:45
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907658/

[6907659] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   00:45:25
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907659/

[6907660] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   02:02:38
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907660/

[6907661] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-workunit/db/postgres}}
time:   01:44:29
info:   http://pulpito.ceph.com/pdonnell-2022-06-30_19:06:15-fs:workload-wip-pdonnell-testing-20220630.004912-distro-default-smithi/6907661/