2023-07-10T22:56:09.492 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-10T22:56:09.493 INFO:teuthology.results:subset: '199/512'
2023-07-10T22:56:09.494 INFO:teuthology.results:seed: '948'
2023-07-11T01:56:07.526 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-11T01:56:07.549 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-11T13:46:34.691 INFO:teuthology.results:Tests finished! gathering results...
2023-07-11T13:46:54.543 INFO:scrape:Found 216 jobs
2023-07-11T13:46:58.147 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332979/teuthology.log
2023-07-11T13:46:58.148 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332979/teuthology.log
2023-07-11T13:47:02.270 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332967/teuthology.log
2023-07-11T13:47:21.051 WARNING:scrape:Misunderstood line: 
2023-07-11T13:47:21.052 INFO:scrape:Found 21 distinct failure reasons
2023-07-11T13:47:21.052 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi084 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'
2023-07-11T13:47:21.052 INFO:scrape:1 jobs: ['7332896']
2023-07-11T13:47:21.053 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}']
2023-07-11T13:47:21.053 INFO:scrape:
2023-07-11T13:47:21.053 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
2023-07-11T13:47:21.054 INFO:scrape:8 jobs: ['7332834', '7332926', '7332958', '7332949', '7333012', '7332894', '7332826', '7332850']
2023-07-11T13:47:21.054 INFO:scrape:suites intersection: ['2-logrotate}']
2023-07-11T13:47:21.054 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/iozone}}', 'begin/{0-install', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/mirror-ha/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{whitelist_health}', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'workloads/cephfs-mirror-ha-workunit}', 'wsync/yes}']
2023-07-11T13:47:21.055 INFO:scrape:
2023-07-11T13:47:21.055 INFO:scrape:Dead: b'2023-07-11T12:11:37.574 DEBUG:teuthology.exit:Finished running handlers'
2023-07-11T13:47:21.055 INFO:scrape:ceph version 18.0.0-4828-gb5891ed6 (b5891ed62e7aba6227aed8ab09a4dd0853a1caf2) reef (dev)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff9bf812cf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff9c14a8e19]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a8f85) [0x7ff9c14a8f85]
 6: (SimpleLock::put_xlock()+0x332) [0x557fdcaf2bc2]
 7: (Locker::xlock_finish(std::_Rb_tree_const_iterator<MutationImpl::LockOp> const&, MutationImpl*, bool*)+0x8c) [0x557fdcadae5c]
 8: (Locker::_drop_locks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*, bool)+0x407) [0x557fdcae3a37]
 9: (Locker::drop_non_rdlocks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*)+0x5f) [0x557fdcae82af]
 10: (Server::reply_client_request(boost::intrusive_ptr<MDRequestImpl>&, boost::intrusive_ptr<MClientReply> const&)+0x497) [0x557fdc93a137]
 11: (Server::respond_to_request(boost::intrusive_ptr<MDRequestImpl>&, int)+0x350) [0x557fdc93afe0]
 12: (Server::_link_local_finish(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CInode*, unsigned long, unsigned long, bool)+0xf5) [0x557fdc93cee5]
 13: (C_MDS_link_local_finish::finish(int)+0x35) [0x557fdc9c3585]
 14: (MDSContext::complete(int)+0x5f) [0x557fdcc2179f]
 15: (MDSIOContextBase::complete(int)+0x524) [0x557fdcc21f24]
 16: (MDSLogContextBase::complete(int)+0x5b) [0x557fdcc222db]
 17: (Finisher::finisher_thread_entry()+0x18d) [0x7ff9c154c93d]
 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff9bf8081ca]
 19: clone()
2023-07-11T13:47:21.056 INFO:scrape:2 jobs: ['7332904', '7332875']
2023-07-11T13:47:21.056 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health']
2023-07-11T13:47:21.056 INFO:scrape:suites union: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/xfstests-dev}', 'tasks/{0-client']
2023-07-11T13:47:21.057 INFO:scrape:
2023-07-11T13:47:21.057 INFO:scrape:Failure: "2023-07-10T23:43:14.716627+0000 mon.a (mon.0) 1530 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)" in cluster log
2023-07-11T13:47:21.057 INFO:scrape:2 jobs: ['7332839', '7332827']
2023-07-11T13:47:21.058 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-07-11T13:47:21.058 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}', 'tasks/metrics}']
2023-07-11T13:47:21.058 INFO:scrape:
2023-07-11T13:47:21.058 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230710.195410
2023-07-11T13:47:21.059 INFO:scrape:11 jobs: ['7332858', '7332847', '7332857', '7332844', '7332830', '7332856', '7332832', '7332860', '7332842', '7332859', '7332848']
2023-07-11T13:47:21.059 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2023-07-11T13:47:21.059 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-mount/mount/fuse', '1-tests/dbench}}}', '2-logrotate}', '2-scrub/no', '2-workload/stress_tests/{0-client-upgrade', '2-workunit/fs/snaps}}', '3-snaps/no', '4-flush/no', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'centos_latest', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a5s-mds-1c-client', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro/{centos_8}', 'frag', 'fs/snaps/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'msgr-failures/osd-mds-delay', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/dbench', 'tasks/workunit/snaps}', 'tasks/{0-from/pacific', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-quota}', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'validater/lockdep}', 'wsync/no}']
2023-07-11T13:47:21.060 INFO:scrape:
2023-07-11T13:47:21.060 INFO:scrape:Failure: {'smithi163.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args': {'executable': None, 'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}}, 'msg': "! Finding LWP::UserAgent::Determined on cpanmetadb failed.\n! cannot open file '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such file or directory opening compressed index\n! Couldn't find module or a distribution LWP::UserAgent::Determined\n! Installing the dependencies failed: Module 'LWP::UserAgent::Determined' is not installed\n! Bailing out the installation for Amazon-S3-0.63.\n"}, 'smithi061.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args': {'executable': None, 'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}}, 'msg': "! Finding Test::More on cpanmetadb failed.\n! cannot open file '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such file or directory opening compressed index\n! Couldn't find module or a distribution Test::More (1.302190)\n! Installing the dependencies failed: Installed version (1.302183) of Test::More is not in range '1.302190'\n! Bailing out the installation for Amazon-S3-0.63.\n"}, 'smithi134.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args': {'executable': None, 'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}}, 'msg': "! Finding XML::Simple on cpanmetadb failed.\n! cannot open file '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such file or directory opening compressed index\n! Couldn't find module or a distribution XML::Simple\n! Installing the dependencies failed: Module 'XML::Simple' is not installed\n! Bailing out the installation for Amazon-S3-0.63.\n"}, 'smithi192.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args': {'executable': None, 'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}}, 'msg': "! Finding Test::Output on cpanmetadb failed.\n! cannot open file '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such file or directory opening compressed index\n! Couldn't find module or a distribution Test::Output (1.033)\n! Installing the dependencies failed: Module 'Test::Output' is not installed\n! Bailing out the installation for Amazon-S3-0.63.\n"}, 'smithi031.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args': {'executable': None, 'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}}, 'msg': "! Finding Digest::MD5::File on cpanmetadb failed.\n! cannot open file '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such file or directory opening compressed index\n! Couldn't find module or a distribution Digest::MD5::File (0.08)\n! Installing the dependencies failed: Module 'Digest::MD5::File' is not installed\n! Bailing out the installation for Amazon-S3-0.63.\n"}}
2023-07-11T13:47:21.060 INFO:scrape:3 jobs: ['7333022', '7332996', '7332968']
2023-07-11T13:47:21.061 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-07-11T13:47:21.061 INFO:scrape:suites union: ['1-ceph', '1-tests/fscrypt-iozone}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/fscrypt/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/mdtest}', 'tasks/volumes/{overrides', 'tasks/{0-client', 'test/snapshot}}']
2023-07-11T13:47:21.061 INFO:scrape:
2023-07-11T13:47:21.062 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-07-11T13:47:21.062 INFO:scrape:1 jobs: ['7332852']
2023-07-11T13:47:21.062 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}']
2023-07-11T13:47:21.063 INFO:scrape:
2023-07-11T13:47:21.063 INFO:scrape:Crash: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh'
2023-07-11T13:47:21.063 INFO:scrape:ceph version 18.0.0-4828-gb5891ed6 (b5891ed62e7aba6227aed8ab09a4dd0853a1caf2) reef (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f4968670420]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x7f4967b1e2b1]
 5: /usr/lib/ceph/libceph-common.so.2(+0x31e413) [0x7f4967b1e413]
 6: (CInode::finish_scatter_gather_update(int, boost::intrusive_ptr<MutationImpl>&)+0x2227) [0x556df6759117]
 7: (Locker::scatter_writebehind(ScatterLock*)+0x19c) [0x556df667642c]
 8: (Locker::simple_sync(SimpleLock*, bool*)+0x358) [0x556df6681098]
 9: (Locker::scatter_nudge(ScatterLock*, MDSContext*, bool)+0x69d) [0x556df6684b9d]
 10: (Locker::scatter_tick()+0x1f4) [0x556df6685454]
 11: (Locker::tick()+0xd) [0x556df6691f2d]
 12: (MDSRankDispatcher::tick()+0x21f) [0x556df64984ef]
 13: (Context::complete(int)+0xd) [0x556df6475fcd]
 14: (CommonSafeTimer<ceph::fair_mutex>::timer_thread()+0x178) [0x7f4967c11328]
 15: (CommonSafeTimerThread<ceph::fair_mutex>::entry()+0x11) [0x7f4967c12801]
 16: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f4968664609]
 17: clone()
2023-07-11T13:47:21.064 INFO:scrape:1 jobs: ['7332814']
2023-07-11T13:47:21.064 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-07-11T13:47:21.064 INFO:scrape:
2023-07-11T13:47:21.065 INFO:scrape:Failure: {'smithi164.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args': {'executable': None, 'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}}, 'msg': "! Finding Amazon::S3 on cpanmetadb failed.\n! cannot open file '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such file or directory opening compressed index\n! Couldn't find module or a distribution Amazon::S3\n"}, 'smithi152.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args': {'executable': None, 'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}}, 'msg': "! Finding Amazon::S3 on cpanmetadb failed.\n! cannot open file '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such file or directory opening compressed index\n! Couldn't find module or a distribution Amazon::S3\n"}}
2023-07-11T13:47:21.065 INFO:scrape:1 jobs: ['7333015']
2023-07-11T13:47:21.065 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_dbench', 'traceless/50pc}']
2023-07-11T13:47:21.066 INFO:scrape:
2023-07-11T13:47:21.066 INFO:scrape:Dead: None
2023-07-11T13:47:21.066 INFO:scrape:2 jobs: ['7332979', '7332967']
2023-07-11T13:47:21.066 INFO:scrape:
2023-07-11T13:47:21.067 INFO:scrape:Failure: Command failed on smithi033 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-4828-gb5891ed6-1focal cephadm=18.0.0-4828-gb5891ed6-1focal ceph-mds=18.0.0-4828-gb5891ed6-1focal ceph-mgr=18.0.0-4828-gb5891ed6-1focal ceph-common=18.0.0-4828-gb5891ed6-1focal ceph-fuse=18.0.0-4828-gb5891ed6-1focal ceph-test=18.0.0-4828-gb5891ed6-1focal ceph-volume=18.0.0-4828-gb5891ed6-1focal radosgw=18.0.0-4828-gb5891ed6-1focal python3-rados=18.0.0-4828-gb5891ed6-1focal python3-rgw=18.0.0-4828-gb5891ed6-1focal python3-cephfs=18.0.0-4828-gb5891ed6-1focal python3-rbd=18.0.0-4828-gb5891ed6-1focal libcephfs2=18.0.0-4828-gb5891ed6-1focal libcephfs-dev=18.0.0-4828-gb5891ed6-1focal librados2=18.0.0-4828-gb5891ed6-1focal librbd1=18.0.0-4828-gb5891ed6-1focal rbd-fuse=18.0.0-4828-gb5891ed6-1focal python3-cephfs=18.0.0-4828-gb5891ed6-1focal cephfs-shell=18.0.0-4828-gb5891ed6-1focal cephfs-top=18.0.0-4828-gb5891ed6-1focal cephfs-mirror=18.0.0-4828-gb5891ed6-1focal'
2023-07-11T13:47:21.067 INFO:scrape:1 jobs: ['7332951']
2023-07-11T13:47:21.067 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2023-07-11T13:47:21.068 INFO:scrape:
2023-07-11T13:47:21.068 INFO:scrape:Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool)
2023-07-11T13:47:21.068 INFO:scrape:1 jobs: ['7332923']
2023-07-11T13:47:21.068 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/alternate-pool}']
2023-07-11T13:47:21.069 INFO:scrape:
2023-07-11T13:47:21.069 INFO:scrape:Dead: b'2023-07-11T13:39:45.311 DEBUG:teuthology.exit:Finished running handlers'
2023-07-11T13:47:21.069 INFO:scrape:2 jobs: ['7333002', '7332937']
2023-07-11T13:47:21.070 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'n/3', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay']
2023-07-11T13:47:21.070 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2023-07-11T13:47:21.070 INFO:scrape:
2023-07-11T13:47:21.070 INFO:scrape:Failure: Command failed on smithi171 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-4828-gb5891ed6-1jammy cephadm=18.0.0-4828-gb5891ed6-1jammy ceph-mds=18.0.0-4828-gb5891ed6-1jammy ceph-mgr=18.0.0-4828-gb5891ed6-1jammy ceph-common=18.0.0-4828-gb5891ed6-1jammy ceph-fuse=18.0.0-4828-gb5891ed6-1jammy ceph-test=18.0.0-4828-gb5891ed6-1jammy ceph-volume=18.0.0-4828-gb5891ed6-1jammy radosgw=18.0.0-4828-gb5891ed6-1jammy python3-rados=18.0.0-4828-gb5891ed6-1jammy python3-rgw=18.0.0-4828-gb5891ed6-1jammy python3-cephfs=18.0.0-4828-gb5891ed6-1jammy python3-rbd=18.0.0-4828-gb5891ed6-1jammy libcephfs2=18.0.0-4828-gb5891ed6-1jammy libcephfs-dev=18.0.0-4828-gb5891ed6-1jammy librados2=18.0.0-4828-gb5891ed6-1jammy librbd1=18.0.0-4828-gb5891ed6-1jammy rbd-fuse=18.0.0-4828-gb5891ed6-1jammy python3-cephfs=18.0.0-4828-gb5891ed6-1jammy cephfs-shell=18.0.0-4828-gb5891ed6-1jammy cephfs-top=18.0.0-4828-gb5891ed6-1jammy cephfs-mirror=18.0.0-4828-gb5891ed6-1jammy'
2023-07-11T13:47:21.071 INFO:scrape:1 jobs: ['7332988']
2023-07-11T13:47:21.071 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}']
2023-07-11T13:47:21.071 INFO:scrape:
2023-07-11T13:47:21.072 INFO:scrape:Failure: Command failed on smithi160 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-07-11T13:47:21.072 INFO:scrape:3 jobs: ['7332876', '7333008', '7332911']
2023-07-11T13:47:21.072 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}']
2023-07-11T13:47:21.073 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distros/ubuntu_latest', 'frag', 'fs/multiclient/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}']
2023-07-11T13:47:21.073 INFO:scrape:
2023-07-11T13:47:21.073 INFO:scrape:Failure: HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4116ff9880>: Failed to establish a new connection: [Errno 110] Connection timed out'))
2023-07-11T13:47:21.073 INFO:scrape:2 jobs: ['7333009', '7333004']
2023-07-11T13:47:21.074 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health']
2023-07-11T13:47:21.074 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds_creation_retry}', 'tasks/metrics}']
2023-07-11T13:47:21.074 INFO:scrape:
2023-07-11T13:47:21.075 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi084 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-07-11T13:47:21.075 INFO:scrape:1 jobs: ['7332984']
2023-07-11T13:47:21.075 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-07-11T13:47:21.076 INFO:scrape:
2023-07-11T13:47:21.076 INFO:scrape:Failure: Test failure: test_subvolume_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc)
2023-07-11T13:47:21.076 INFO:scrape:1 jobs: ['7332969']
2023-07-11T13:47:21.076 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/misc}}']
2023-07-11T13:47:21.077 INFO:scrape:
2023-07-11T13:47:21.077 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2023-07-11T13:47:21.077 INFO:scrape:1 jobs: ['7332956']
2023-07-11T13:47:21.078 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2023-07-11T13:47:21.078 INFO:scrape:
2023-07-11T13:47:21.078 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'
2023-07-11T13:47:21.078 INFO:scrape:1 jobs: ['7332908']
2023-07-11T13:47:21.079 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-07-11T13:47:21.079 INFO:scrape:
2023-07-11T13:47:21.079 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable)
2023-07-11T13:47:21.080 INFO:scrape:1 jobs: ['7332887']
2023-07-11T13:47:21.080 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}']
2023-07-11T13:47:21.080 INFO:scrape:
2023-07-11T13:47:21.092 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/
failed:  37
dead:    10
running: 0
waiting: 0
queued:  0
passed:  169


Fail
=================================================================
[7332814]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:41:23
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332814/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332814/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b8d1384ab897432491e7659fc23ec4b3

    Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi008 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh'

[7332826]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   01:05:15
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332826/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332826/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7332827]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:32:41
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332827/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332827/

    "2023-07-10T23:38:49.755180+0000 mon.a (mon.0) 2344 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg inactive, 1 pg peering
    (PG_AVAILABILITY)" in cluster log

[7332830]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:21:44
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332830/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332830/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b27b6076cce44d30a42e6e72696080a2

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b58
    91ed62e7aba6227aed8ab09a4dd0853a1caf2

[7332832]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}}
-----------------------------------------------------------------
time:   00:21:07
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332832/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332832/
sentry: https://sentry.ceph.com/organizations/ceph/?query=14f3f3b76b154cc9bb341969e5fc3823

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b58
    91ed62e7aba6227aed8ab09a4dd0853a1caf2

[7332834]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
-----------------------------------------------------------------
time:   01:27:40
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332834/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332834/

    error during scrub thrashing: reached maximum tries (31) after waiting for
    900 seconds

[7332839]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   01:06:08
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332839/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332839/

    "2023-07-10T23:43:14.716627+0000 mon.a (mon.0) 1530 : cluster [WRN] Health
    check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)"
    in cluster log

[7332842]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:07:24
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332842/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332842/
sentry: https://sentry.ceph.com/organizations/ceph/?query=78ed8fa896bf44d7b853ec525f06faa0

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b58
    91ed62e7aba6227aed8ab09a4dd0853a1caf2

[7332844]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:12:18
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332844/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332844/
sentry: https://sentry.ceph.com/organizations/ceph/?query=19246a81b6d04b2e94aca42e12662f9e

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b58
    91ed62e7aba6227aed8ab09a4dd0853a1caf2

[7332847]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep}
-----------------------------------------------------------------
time:   00:09:50
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332847/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332847/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1851782b1a394361972c6eb8fd2cdaff

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b58
    91ed62e7aba6227aed8ab09a4dd0853a1caf2

[7332848]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:09:53
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332848/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332848/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9e85015e7eb045d9910143db1775241d

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b58
    91ed62e7aba6227aed8ab09a4dd0853a1caf2

[7332850]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   01:13:42
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332850/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332850/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7332852]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   00:20:27
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332852/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332852/
sentry: https://sentry.ceph.com/organizations/ceph/?query=965741ee97aa4c878ceeeaff2fa78908

    Test failure: test_disable_enable_human_readable_quota_values
    (tasks.cephfs.test_quota.TestQuota)

[7332856]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332856/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332856/
sentry: https://sentry.ceph.com/organizations/ceph/?query=71679159dbc9407ab5fc059ad6951198

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-
    pdonnell-testing-20230710.195410

[7332857]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332857/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332857/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f7f615f712c34b85a44c3d91a181d8a6

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-
    pdonnell-testing-20230710.195410

[7332858]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332858/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332858/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0efd0f47bb9e45c39ec7e4cafba9ed14

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-
    pdonnell-testing-20230710.195410

[7332859]  fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332859/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332859/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d03bbe2e291941cea932df21a20d6934

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-
    pdonnell-testing-20230710.195410

[7332860]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332860/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332860/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6ef47db7230e4566bd8b9ae09d9aaa1d

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-
    pdonnell-testing-20230710.195410

[7332876]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:16:11
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332876/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332876/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c0ab378c34b74baca70faea2be9822b7

    Command failed on smithi160 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7332887]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:58:12
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332887/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332887/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0b7d120dcf394005bc0447f833996374

    saw valgrind issues

[7332894]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:39:07
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332894/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332894/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7332896]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   00:12:34
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332896/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332896/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5a5f4791dd6c4eec819b90064389cf96

    Command failed (workunit test fs/test_python.sh) on smithi084 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'

[7332908]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:15:30
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332908/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332908/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fc89092b527d4c7b9d18736cf1e6daae

    Command failed (workunit test fs/quota/quota.sh) on smithi150 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'

[7332911]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:14:45
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332911/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332911/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ee75b18031934b38874c7ebef410d271

    Command failed on smithi083 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7332923]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
-----------------------------------------------------------------
time:   00:19:04
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332923/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332923/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cdccbca5be1e4536882334d6f923a14a

    Test failure: test_rebuild_simple
    (tasks.cephfs.test_recovery_pool.TestRecoveryPool)

[7332926]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:37:58
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332926/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332926/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7332949]  fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_20.04} workloads/cephfs-mirror-ha-workunit}
-----------------------------------------------------------------
time:   00:31:36
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332949/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332949/

    reached maximum tries (51) after waiting for 300 seconds

[7332951]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:07:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332951/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332951/
sentry: https://sentry.ceph.com/organizations/ceph/?query=104c1be3ab3344ae9b1b594f9c3ccdb7

    Command failed on smithi033 with status 100: 'sudo
    DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="
    --force-confdef" -o Dpkg::Options::="--force-confold" install
    ceph=18.0.0-4828-gb5891ed6-1focal cephadm=18.0.0-4828-gb5891ed6-1focal
    ceph-mds=18.0.0-4828-gb5891ed6-1focal ceph-mgr=18.0.0-4828-gb5891ed6-1focal
    ceph-common=18.0.0-4828-gb5891ed6-1focal ceph-
    fuse=18.0.0-4828-gb5891ed6-1focal ceph-test=18.0.0-4828-gb5891ed6-1focal
    ceph-volume=18.0.0-4828-gb5891ed6-1focal
    radosgw=18.0.0-4828-gb5891ed6-1focal
    python3-rados=18.0.0-4828-gb5891ed6-1focal
    python3-rgw=18.0.0-4828-gb5891ed6-1focal
    python3-cephfs=18.0.0-4828-gb5891ed6-1focal
    python3-rbd=18.0.0-4828-gb5891ed6-1focal
    libcephfs2=18.0.0-4828-gb5891ed6-1focal libcephfs-
    dev=18.0.0-4828-gb5891ed6-1focal librados2=18.0.0-4828-gb5891ed6-1focal
    librbd1=18.0.0-4828-gb5891ed6-1focal rbd-fuse=18.0.0-4828-gb5891ed6-1focal
    python3-cephfs=18.0.0-4828-gb5891ed6-1focal cephfs-
    shell=18.0.0-4828-gb5891ed6-1focal cephfs-top=18.0.0-4828-gb5891ed6-1focal
    cephfs-mirror=18.0.0-4828-gb5891ed6-1focal'

[7332956]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   01:23:19
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332956/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332956/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7372f8be4df1419996c6feb61fefa897

    Test failure: test_cephfs_mirror_restart_sync_on_blocklist
    (tasks.cephfs.test_mirroring.TestMirroring)

[7332958]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:35:58
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332958/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332958/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7332969]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:24:50
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332969/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332969/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c951e543c77c4d14883a13a64588734e

    Test failure: test_subvolume_upgrade_v1_to_v2
    (tasks.cephfs.test_volumes.TestMisc)

[7332984]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:35:05
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332984/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332984/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2146f76d42f14f3ea1a35fd0dc5c3b67

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi084
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7332988]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:09:56
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332988/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332988/
sentry: https://sentry.ceph.com/organizations/ceph/?query=966a2f95dde84faeb354c9e3cab27393

    Command failed on smithi171 with status 100: 'sudo
    DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="
    --force-confdef" -o Dpkg::Options::="--force-confold" install
    ceph=18.0.0-4828-gb5891ed6-1jammy cephadm=18.0.0-4828-gb5891ed6-1jammy
    ceph-mds=18.0.0-4828-gb5891ed6-1jammy ceph-mgr=18.0.0-4828-gb5891ed6-1jammy
    ceph-common=18.0.0-4828-gb5891ed6-1jammy ceph-
    fuse=18.0.0-4828-gb5891ed6-1jammy ceph-test=18.0.0-4828-gb5891ed6-1jammy
    ceph-volume=18.0.0-4828-gb5891ed6-1jammy
    radosgw=18.0.0-4828-gb5891ed6-1jammy
    python3-rados=18.0.0-4828-gb5891ed6-1jammy
    python3-rgw=18.0.0-4828-gb5891ed6-1jammy
    python3-cephfs=18.0.0-4828-gb5891ed6-1jammy
    python3-rbd=18.0.0-4828-gb5891ed6-1jammy
    libcephfs2=18.0.0-4828-gb5891ed6-1jammy libcephfs-
    dev=18.0.0-4828-gb5891ed6-1jammy librados2=18.0.0-4828-gb5891ed6-1jammy
    librbd1=18.0.0-4828-gb5891ed6-1jammy rbd-fuse=18.0.0-4828-gb5891ed6-1jammy
    python3-cephfs=18.0.0-4828-gb5891ed6-1jammy cephfs-
    shell=18.0.0-4828-gb5891ed6-1jammy cephfs-top=18.0.0-4828-gb5891ed6-1jammy
    cephfs-mirror=18.0.0-4828-gb5891ed6-1jammy'

[7333004]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
-----------------------------------------------------------------
time:   00:16:32
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333004/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333004/
sentry: https://sentry.ceph.com/organizations/ceph/?query=52e68a21015a49f3983731891ce0c941

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent
    os%2F8%2Fx86_64&sha1=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2 (Caused by
    NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f21327e1a90>: Failed to establish a new connection: [Errno 110]
    Connection timed out'))

[7333008]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   00:17:27
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333008/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333008/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5eb02036428945f684a66d61dcf14739

    Command failed on smithi055 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay-quay-
    quay.apps.os.sepia.ceph.com/ceph-
    ci/ceph:b5891ed62e7aba6227aed8ab09a4dd0853a1caf2 pull'

[7333009]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:14:18
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333009/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333009/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d98508456afd40a188bc6cda62ab4acf

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent
    os%2F8%2Fx86_64&sha1=b5891ed62e7aba6227aed8ab09a4dd0853a1caf2 (Caused by
    NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f4116ff9880>: Failed to establish a new connection: [Errno 110]
    Connection timed out'))

[7333012]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}}
-----------------------------------------------------------------
time:   01:06:48
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333012/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333012/

    error during scrub thrashing: reached maximum tries (31) after waiting for
    900 seconds



Dead
=================================================================
[7332875]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332875/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332875/

    hit max job timeout

[7332904]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332904/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332904/

    hit max job timeout

[7332937]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332937/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332937/

    hit max job timeout

[7332967]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332967/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332967/

    Error reimaging machines: Timeout opening channel.

[7332968]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}}
-----------------------------------------------------------------
time:   00:08:55
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332968/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332968/
sentry: https://sentry.ceph.com/organizations/ceph/?query=711071e115ec4560840dc449c47f68a8

    {'smithi178.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args':
    {'executable': None, 'from_path': None, 'installdeps': False, 'locallib':
    None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest':
    False, 'system_lib': False, 'version': None}}, 'msg': "! Finding
    XML::Simple on cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution XML::Simple\n! Installing the dependencies failed: Module
    'XML::Simple' is not installed\n! Bailing out the installation for
    Amazon-S3-0.63.\n"}, 'smithi045.front.sepia.ceph.com': {'_ansible_no_log':
    False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation':
    {'module_args': {'executable': None, 'from_path': None, 'installdeps':
    False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name':
    'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}},
    'msg': "! Finding Class::Accessor::Fast on cpanmetadb failed.\n! cannot
    open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Class::Accessor::Fast\n! Finding Net::Amazon::Signature::V4 on
    cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Net::Amazon::Signature::V4\n! Installing the dependencies
    failed: Module 'Net::Amazon::Signature::V4' is not installed, Module
    'Class::Accessor::Fast' is not installed\n! Bailing out the installation
    for Amazon-S3-0.63.\n"}, 'smithi072.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm
    Amazon::S3', 'invocation': {'module_args': {'executable': None,
    'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None,
    'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib':
    False, 'version': None}}, 'msg': "! Finding Net::Amazon::Signature::V4 on
    cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Net::Amazon::Signature::V4\n! Finding Class::Accessor::Fast on
    cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Class::Accessor::Fast\n! Finding Digest::MD5::File on
    cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Digest::MD5::File (0.08)\n! Installing the dependencies
    failed: Module 'Class::Accessor::Fast' is not installed, Module
    'Digest::MD5::File' is not installed, Module 'Net::Amazon::Signature::V4'
    is not installed\n! Bailing out the installation for Amazon-S3-0.63.\n"}}

[7332979]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332979/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332979/

    Error reimaging machines: Timeout opening channel.

[7332996]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
-----------------------------------------------------------------
time:   00:07:47
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332996/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332996/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6d7e396340b645a089f634e5784b174b

    {'smithi040.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args':
    {'executable': None, 'from_path': None, 'installdeps': False, 'locallib':
    None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest':
    False, 'system_lib': False, 'version': None}}, 'msg': "! Finding
    Test::Output on cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Test::Output (1.033)\n! Installing the dependencies failed:
    Module 'Test::Output' is not installed\n! Bailing out the installation for
    Amazon-S3-0.63.\n"}, 'smithi008.front.sepia.ceph.com': {'_ansible_no_log':
    False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation':
    {'module_args': {'executable': None, 'from_path': None, 'installdeps':
    False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name':
    'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}},
    'msg': "! Finding Test::Output on cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Test::Output (1.033)\n! Installing the dependencies failed:
    Module 'Test::Output' is not installed\n! Bailing out the installation for
    Amazon-S3-0.63.\n"}}

[7333002]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333002/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333002/

    hit max job timeout

[7333015]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
-----------------------------------------------------------------
time:   00:05:55
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333015/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333015/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b0e6e625a9494877a579ebc2295d3198

    {'smithi164.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args':
    {'executable': None, 'from_path': None, 'installdeps': False, 'locallib':
    None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest':
    False, 'system_lib': False, 'version': None}}, 'msg': "! Finding Amazon::S3
    on cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Amazon::S3\n"}, 'smithi152.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm
    Amazon::S3', 'invocation': {'module_args': {'executable': None,
    'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None,
    'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib':
    False, 'version': None}}, 'msg': "! Finding Amazon::S3 on cpanmetadb
    failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Amazon::S3\n"}}

[7333022]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:08:10
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333022/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333022/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f88f70fec8084db3ad52e01882de98d2

    {'smithi163.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args':
    {'executable': None, 'from_path': None, 'installdeps': False, 'locallib':
    None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest':
    False, 'system_lib': False, 'version': None}}, 'msg': "! Finding
    LWP::UserAgent::Determined on cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution LWP::UserAgent::Determined\n! Installing the dependencies
    failed: Module 'LWP::UserAgent::Determined' is not installed\n! Bailing out
    the installation for Amazon-S3-0.63.\n"}, 'smithi061.front.sepia.ceph.com':
    {'_ansible_no_log': False, 'changed': False, 'cmd': '/usr/bin/cpanm
    Amazon::S3', 'invocation': {'module_args': {'executable': None,
    'from_path': None, 'installdeps': False, 'locallib': None, 'mirror': None,
    'mirror_only': False, 'name': 'Amazon::S3', 'notest': False, 'system_lib':
    False, 'version': None}}, 'msg': "! Finding Test::More on cpanmetadb
    failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Test::More (1.302190)\n! Installing the dependencies failed:
    Installed version (1.302183) of Test::More is not in range '1.302190'\n!
    Bailing out the installation for Amazon-S3-0.63.\n"},
    'smithi134.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation': {'module_args':
    {'executable': None, 'from_path': None, 'installdeps': False, 'locallib':
    None, 'mirror': None, 'mirror_only': False, 'name': 'Amazon::S3', 'notest':
    False, 'system_lib': False, 'version': None}}, 'msg': "! Finding
    XML::Simple on cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution XML::Simple\n! Installing the dependencies failed: Module
    'XML::Simple' is not installed\n! Bailing out the installation for
    Amazon-S3-0.63.\n"}, 'smithi192.front.sepia.ceph.com': {'_ansible_no_log':
    False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation':
    {'module_args': {'executable': None, 'from_path': None, 'installdeps':
    False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name':
    'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}},
    'msg': "! Finding Test::Output on cpanmetadb failed.\n! cannot open file
    '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such
    file or directory opening compressed index\n! Couldn't find module or a
    distribution Test::Output (1.033)\n! Installing the dependencies failed:
    Module 'Test::Output' is not installed\n! Bailing out the installation for
    Amazon-S3-0.63.\n"}, 'smithi031.front.sepia.ceph.com': {'_ansible_no_log':
    False, 'changed': False, 'cmd': '/usr/bin/cpanm Amazon::S3', 'invocation':
    {'module_args': {'executable': None, 'from_path': None, 'installdeps':
    False, 'locallib': None, 'mirror': None, 'mirror_only': False, 'name':
    'Amazon::S3', 'notest': False, 'system_lib': False, 'version': None}},
    'msg': "! Finding Digest::MD5::File on cpanmetadb failed.\n! cannot open
    file '/root/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No
    such file or directory opening compressed index\n! Couldn't find module or
    a distribution Digest::MD5::File (0.08)\n! Installing the dependencies
    failed: Module 'Digest::MD5::File' is not installed\n! Bailing out the
    installation for Amazon-S3-0.63.\n"}}



Pass
=================================================================
[7332808] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:26:53
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332808/

[7332809] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}}
time:   00:45:48
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332809/

[7332810] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:22:06
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332810/

[7332811] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:23:45
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332811/

[7332812] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:34:57
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332812/

[7332813] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:28:31
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332813/

[7332815] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}}
time:   00:32:40
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332815/

[7332816] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:17:03
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332816/

[7332817] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:36
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332817/

[7332818] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:30
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332818/

[7332819] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:16:25
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332819/

[7332820] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:22:06
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332820/

[7332821] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:21:16
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332821/

[7332822] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}}
time:   00:34:26
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332822/

[7332823] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:21:09
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332823/

[7332824] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:28:25
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332824/

[7332825] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:21:50
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332825/

[7332828] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:29:38
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332828/

[7332829] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:23:39
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332829/

[7332831] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:21:55
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332831/

[7332833] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:19:14
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332833/

[7332835] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:22:57
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332835/

[7332836] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:45
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332836/

[7332837] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health}
time:   00:24:32
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332837/

[7332838] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:32:16
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332838/

[7332840] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:52:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332840/

[7332841] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:14
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332841/

[7332843] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:18:25
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332843/

[7332845] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
time:   00:15:32
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332845/

[7332846] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:28:59
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332846/

[7332849] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:35:07
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332849/

[7332851] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:12:51
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332851/

[7332853] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:53:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332853/

[7332854] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:50:20
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332854/

[7332855] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:31:48
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332855/

[7332861] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}}
time:   00:39:57
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332861/

[7332862] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:37:29
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332862/

[7332863] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}}
time:   00:17:54
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332863/

[7332864] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
time:   00:17:35
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332864/

[7332865] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}}
time:   00:31:54
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332865/

[7332866] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:35:44
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332866/

[7332867] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:29:37
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332867/

[7332868] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:39
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332868/

[7332869] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:17:32
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332869/

[7332870] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
time:   00:52:07
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332870/

[7332871] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}}
time:   00:38:57
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332871/

[7332872] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:31:09
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332872/

[7332873] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:32:31
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332873/

[7332874] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}
time:   00:23:30
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332874/

[7332877] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:32:20
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332877/

[7332878] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:36:14
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332878/

[7332879] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
time:   01:48:22
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332879/

[7332880] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}}
time:   00:30:35
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332880/

[7332881] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:17:11
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332881/

[7332882] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:35:22
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332882/

[7332883] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:24:21
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332883/

[7332884] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}}
time:   00:36:03
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332884/

[7332885] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:41:58
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332885/

[7332886] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:20:22
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332886/

[7332888] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:20:51
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332888/

[7332889] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}}
time:   00:41:09
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332889/

[7332890] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:18:07
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332890/

[7332891] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:29:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332891/

[7332892] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:37:32
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332892/

[7332893] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:14:02
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332893/

[7332895] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:07
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332895/

[7332897] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:45
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332897/

[7332898] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   00:55:44
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332898/

[7332899] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}}
time:   00:26:13
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332899/

[7332900] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:19:14
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332900/

[7332901] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
time:   00:32:08
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332901/

[7332902] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:32:41
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332902/

[7332903] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:19:25
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332903/

[7332905] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   00:53:50
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332905/

[7332906] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}}
time:   00:34:53
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332906/

[7332907] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:27:17
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332907/

[7332909] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:32:31
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332909/

[7332910] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}}
time:   00:48:30
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332910/

[7332912] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:53:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332912/

[7332913] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:53:50
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332913/

[7332914] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:15:25
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332914/

[7332915] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   02:09:57
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332915/

[7332916] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}}
time:   01:48:58
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332916/

[7332917] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:05:59
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332917/

[7332918] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:19:27
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332918/

[7332919] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:21:40
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332919/

[7332920] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:34
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332920/

[7332921] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:17:47
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332921/

[7332922] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}}
time:   00:53:02
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332922/

[7332924] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:33:39
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332924/

[7332925] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
time:   00:26:09
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332925/

[7332927] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep}
time:   00:59:12
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332927/

[7332928] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:24:22
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332928/

[7332929] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:49:25
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332929/

[7332930] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:17:11
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332930/

[7332931] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
time:   00:17:24
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332931/

[7332932] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:14:32
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332932/

[7332933] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}}
time:   01:04:27
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332933/

[7332934] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:16:27
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332934/

[7332935] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
time:   00:15:29
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332935/

[7332936] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:26:16
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332936/

[7332938] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
time:   00:12:06
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332938/

[7332939] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:18:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332939/

[7332940] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}}
time:   00:31:11
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332940/

[7332941] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:30:55
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332941/

[7332942] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
time:   00:21:17
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332942/

[7332943] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:18:44
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332943/

[7332944] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   00:16:16
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332944/

[7332945] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
time:   00:48:21
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332945/

[7332946] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   00:22:58
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332946/

[7332947] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   00:14:25
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332947/

[7332948] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror}
time:   02:03:48
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332948/

[7332950] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:06:42
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332950/

[7332952] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:15:42
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332952/

[7332953] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
time:   00:43:58
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332953/

[7332954] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:55:53
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332954/

[7332955] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop}
time:   00:15:56
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332955/

[7332957] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   00:36:26
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332957/

[7332959] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:33:47
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332959/

[7332960] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:16:26
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332960/

[7332961] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:17:44
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332961/

[7332962] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}
time:   00:13:40
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332962/

[7332963] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}}
time:   00:36:59
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332963/

[7332964] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:55:25
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332964/

[7332965] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:18:41
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332965/

[7332966] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}}
time:   00:17:55
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332966/

[7332970] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}}
time:   00:41:45
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332970/

[7332971] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:21:00
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332971/

[7332972] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
time:   00:53:26
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332972/

[7332973] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}}
time:   00:40:46
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332973/

[7332974] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
time:   00:53:49
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332974/

[7332975] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:46:12
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332975/

[7332976] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   00:51:21
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332976/

[7332977] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:26:57
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332977/

[7332978] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep}
time:   00:43:46
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332978/

[7332980] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:48
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332980/

[7332981] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:25
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332981/

[7332982] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:28:47
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332982/

[7332983] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
time:   00:18:47
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332983/

[7332985] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}}
time:   00:32:47
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332985/

[7332986] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
time:   00:31:18
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332986/

[7332987] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:54:26
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332987/

[7332989] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:52:40
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332989/

[7332990] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:52:23
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332990/

[7332991] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}}
time:   00:21:52
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332991/

[7332992] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}}
time:   00:38:51
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332992/

[7332993] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
time:   00:23:38
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332993/

[7332994] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:44:12
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332994/

[7332995] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:25:14
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332995/

[7332997] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}}
time:   00:41:39
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332997/

[7332998] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
time:   00:15:10
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332998/

[7332999] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}}
time:   00:25:15
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7332999/

[7333000] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:22:50
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333000/

[7333001] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
time:   00:26:20
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333001/

[7333003] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:19:01
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333003/

[7333005] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:23
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333005/

[7333006] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
time:   00:13:42
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333006/

[7333007] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:15:15
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333007/

[7333010] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:21:19
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333010/

[7333011] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:32:52
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333011/

[7333013] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:22:18
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333013/

[7333014] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:36:57
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333014/

[7333016] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:24:56
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333016/

[7333017] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}}
time:   00:43:47
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333017/

[7333018] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
time:   01:46:27
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333018/

[7333019] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:16:23
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333019/

[7333020] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}}
time:   01:12:01
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333020/

[7333021] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:19:46
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333021/

[7333023] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
time:   00:27:21
info:   http://pulpito.ceph.com/pdonnell-2023-07-10_22:55:54-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7333023/