2023-11-04T22:34:41.523 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-04T22:34:41.524 INFO:teuthology.results:subset: '111/12000' 2023-11-04T22:34:41.524 INFO:teuthology.results:seed: '7244' 2023-11-05T01:51:35.204 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-05T01:51:35.237 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-05T12:12:29.206 INFO:teuthology.results:Tests finished! gathering results... 2023-11-05T12:12:46.209 INFO:scrape:Found 194 jobs 2023-11-05T12:12:49.103 WARNING:scrape:Misunderstood line: 2023-11-05T12:12:55.046 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447334/teuthology.log 2023-11-05T12:12:59.189 WARNING:scrape:Misunderstood line: 2023-11-05T12:12:59.650 WARNING:scrape:Misunderstood line: 2023-11-05T12:13:04.084 INFO:scrape:Found 11 distinct failure reasons 2023-11-05T12:13:04.085 INFO:scrape:Failure: "2023-11-04T23:17:26.122309+0000 mon.a (mon.0) 146 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-11-05T12:13:04.085 INFO:scrape:30 jobs 2023-11-05T12:13:04.086 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2023-11-05T12:13:04.087 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'centos_latest', 'clusters/1-mds-1-client', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/admin}', 'tasks/asok_dump_tree}', 'tasks/cap-flush}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-limits}', 'tasks/failover}', 'tasks/fscrypt-dbench}', 'tasks/fscrypt-ffsb}', 'tasks/fscrypt-iozone}', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/multimds_misc}', 'tasks/pool-perm}', 'tasks/quota}', 'tasks/recovery-fs}', 'tasks/scrub}', 'tasks/strays}', 'tasks/trim-i24137}', 'tasks/truncate_delay}', 'tasks/volumes/{overrides', 'test/clone}}', 'test/misc}}', 'test/snapshot}}', 'traceless/50pc}'] 2023-11-05T12:13:04.087 INFO:scrape: 2023-11-05T12:13:04.088 INFO:scrape:Dead: b'2023-11-05T12:09:47.896 DEBUG:teuthology.exit:Finished running handlers' 2023-11-05T12:13:04.088 INFO:scrape:1 jobs: ['7447391'] 2023-11-05T12:13:04.089 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}'] 2023-11-05T12:13:04.089 INFO:scrape: 2023-11-05T12:13:04.090 INFO:scrape:Failure: Command failed on smithi033 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-11-05T12:13:04.090 INFO:scrape:3 jobs: ['7447361', '7447356', '7447426'] 2023-11-05T12:13:04.091 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-11-05T12:13:04.091 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-workunit/suites/ffsb}}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/fscrypt-pjd}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-11-05T12:13:04.092 INFO:scrape: 2023-11-05T12:13:04.092 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-11-05T12:13:04.093 INFO:scrape:4 jobs: ['7447457', '7447439', '7447344', '7447384'] 2023-11-05T12:13:04.093 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay', 'wsync/no}'] 2023-11-05T12:13:04.094 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/norstats}}', '5-workunit/postgres}}', '5-workunit/suites/fsx}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-11-05T12:13:04.094 INFO:scrape: 2023-11-05T12:13:04.095 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-11-05T12:13:04.095 INFO:scrape:2 jobs: ['7447312', '7447383'] 2023-11-05T12:13:04.095 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-11-05T12:13:04.096 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-11-05T12:13:04.096 INFO:scrape: 2023-11-05T12:13:04.097 INFO:scrape:Dead: None 2023-11-05T12:13:04.097 INFO:scrape:2 jobs: ['7447334', '7447354'] 2023-11-05T12:13:04.098 INFO:scrape: 2023-11-05T12:13:04.098 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi045 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=956958b278ef8205ee4c9ec42773c7f763ef5eac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-11-05T12:13:04.099 INFO:scrape:1 jobs: ['7447299'] 2023-11-05T12:13:04.099 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-11-05T12:13:04.099 INFO:scrape: 2023-11-05T12:13:04.100 INFO:scrape:Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 2023-11-05T12:13:04.100 INFO:scrape:1 jobs: ['7447400'] 2023-11-05T12:13:04.101 INFO:scrape:suites: ['fs/nfs/{cluster/{1-node}', 'overrides/ignorelist_health', 'supported-random-distros$/{ubuntu_20.04}', 'tasks/nfs}'] 2023-11-05T12:13:04.101 INFO:scrape: 2023-11-05T12:13:04.102 INFO:scrape:Failure: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) 2023-11-05T12:13:04.102 INFO:scrape:1 jobs: ['7447407'] 2023-11-05T12:13:04.103 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-11-05T12:13:04.103 INFO:scrape: 2023-11-05T12:13:04.104 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi132 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=956958b278ef8205ee4c9ec42773c7f763ef5eac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-11-05T12:13:04.104 INFO:scrape:1 jobs: ['7447436'] 2023-11-05T12:13:04.105 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-11-05T12:13:04.105 INFO:scrape: 2023-11-05T12:13:04.105 INFO:scrape:Crash: "2023-11-05T00:51:23.808829+0000 mon.a (mon.0) 683 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-11-05T12:13:04.106 INFO:scrape:ceph version 18.2.0-1158-g956958b2 (956958b278ef8205ee4c9ec42773c7f763ef5eac) reef (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0487505520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x189) [0x7f0487bdef08] 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x55cbe7211f41] 7: (CDentry::check_corruption(bool)+0x419) [0x55cbe73f7809] 8: ceph-mds(+0x33c45c) [0x55cbe740745c] 9: (CDir::_omap_commit(int)+0x94b) [0x55cbe741016b] 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x132) [0x55cbe7406902] 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x9af) [0x55cbe74d8e2f] 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x55cbe749b162] 13: (MDLog::trim_all()+0x781) [0x55cbe749c811] 14: ceph-mds(+0x17e0b7) [0x55cbe72490b7] 15: ceph-mds(+0x12049d) [0x55cbe71eb49d] 16: (MDSContext::complete(int)+0x5c) [0x55cbe748b1dc] 17: ceph-mds(+0x12a872) [0x55cbe71f5872] 18: (MDSContext::complete(int)+0x5c) [0x55cbe748b1dc] 19: (MDSIOContextBase::complete(int)+0x30c) [0x55cbe748b60c] 20: (Finisher::finisher_thread_entry()+0x175) [0x7f0487c91b25] 21: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0487557b43] 22: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f04875e9a00] 2023-11-05T12:13:04.106 INFO:scrape:1 jobs: ['7447417'] 2023-11-05T12:13:04.107 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/damage}'] 2023-11-05T12:13:04.107 INFO:scrape: 2023-11-05T12:13:04.117 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/ failed: 44 dead: 3 running: 0 waiting: 0 queued: 0 passed: 147 Fail ================================================================= [7447295] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:22:11 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447295/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447295/ "2023-11-04T22:57:46.661658+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447299] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:29:12 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447299/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447299/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c95c17b5b2fc4c85a5ddcedd2829e5b8 Command failed (workunit test libcephfs/test.sh) on smithi045 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=956958b278ef8205ee4c9ec42773c7f763ef5eac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7447301] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:50:48 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447301/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447301/ "2023-11-04T23:00:07.680296+0000 mon.a (mon.0) 121 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447311] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:16:11 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447311/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447311/ "2023-11-04T23:07:56.307339+0000 mon.a (mon.0) 142 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447312] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:27:16 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447312/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447312/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b4b130e77f94701a69ad6711b50c48e saw valgrind issues [7447315] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:14:02 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447315/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447315/ "2023-11-04T23:09:16.959889+0000 mon.a (mon.0) 153 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447319] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} ----------------------------------------------------------------- time: 02:12:25 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447319/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447319/ "2023-11-04T23:12:18.992368+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447325] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 00:40:22 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447325/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447325/ "2023-11-04T23:17:26.122309+0000 mon.a (mon.0) 146 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447332] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447332/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447332/ "2023-11-04T23:26:37.765906+0000 mon.a (mon.0) 124 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447340] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:32:06 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447340/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447340/ "2023-11-04T23:31:09.527090+0000 mon.a (mon.0) 334 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447344] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:39:43 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447344/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447344/ error during scrub thrashing: rank damage found: {'backtrace'} [7447351] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:15:42 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447351/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447351/ "2023-11-04T23:46:16.832161+0000 mon.a (mon.0) 217 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447356] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} ----------------------------------------------------------------- time: 00:18:46 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447356/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447356/ Command failed on smithi148 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' [7447361] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:25:18 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447361/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447361/ Command failed on smithi033 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' [7447363] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:17:20 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447363/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447363/ "2023-11-04T23:56:40.942426+0000 mon.a (mon.0) 194 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447368] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 01:14:13 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447368/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447368/ "2023-11-05T00:02:55.987518+0000 mon.a (mon.0) 144 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447374] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:24:45 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447374/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447374/ "2023-11-05T00:15:49.302136+0000 mon.a (mon.0) 142 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447383] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:26:18 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447383/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50af41680603438f9d532cbc79d3a1e9 saw valgrind issues [7447384] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:53:47 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447384/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447384/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7447386] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} ----------------------------------------------------------------- time: 00:16:04 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447386/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447386/ "2023-11-05T00:17:14.779298+0000 mon.a (mon.0) 164 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447389] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:11:37 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447389/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447389/ "2023-11-05T00:13:57.627388+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447393] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:33:03 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447393/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447393/ "2023-11-05T00:26:10.060338+0000 mon.a (mon.0) 432 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447400] fs/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{ubuntu_20.04} tasks/nfs} ----------------------------------------------------------------- time: 00:19:35 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447400/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447400/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53c484d61cc24b799e1faccb2f9ccdcb Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) [7447401] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:12:58 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447401/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447401/ "2023-11-05T00:28:36.314673+0000 mon.a (mon.0) 233 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447407] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:15:20 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447407/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447407/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1af428ea200749fdb3cef209046dfda2 Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) [7447416] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:22:23 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447416/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447416/ "2023-11-05T00:47:26.574776+0000 mon.a (mon.0) 140 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447417] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:26:48 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447417/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447417/ "2023-11-05T00:51:23.808829+0000 mon.a (mon.0) 683 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447424] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:12:26 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447424/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447424/ "2023-11-05T00:51:34.921502+0000 mon.a (mon.0) 162 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447426] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:14:22 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447426/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=575c305a09c34f2385c4c7ffac3722c0 Command failed on smithi163 with status 5: 'sudo systemctl stop ceph-e91021e0-7b75-11ee-8db9-212e2dc638e7@mon.b' [7447431] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:46 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447431/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447431/ "2023-11-05T01:08:34.702412+0000 mon.a (mon.0) 120 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447436] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:32:38 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447436/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447436/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ae5a1d6550e40dba003e5dba0411511 Command failed (workunit test kernel_untar_build.sh) on smithi132 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=956958b278ef8205ee4c9ec42773c7f763ef5eac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7447437] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 00:48:38 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447437/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447437/ "2023-11-05T01:12:03.379172+0000 mon.a (mon.0) 109 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447439] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:33:07 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447439/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447439/ error during scrub thrashing: rank damage found: {'backtrace'} [7447441] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 01:13:42 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447441/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447441/ "2023-11-05T01:19:21.336555+0000 mon.a (mon.0) 150 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447442] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:29:22 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447442/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447442/ "2023-11-05T01:16:15.325906+0000 mon.a (mon.0) 524 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447448] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:13:55 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447448/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447448/ "2023-11-05T01:20:06.561472+0000 mon.a (mon.0) 161 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447450] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:16:59 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447450/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447450/ "2023-11-05T01:25:49.728841+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447457] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:35:30 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447457/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447457/ error during scrub thrashing: rank damage found: {'backtrace'} [7447461] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:19:52 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447461/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447461/ "2023-11-05T01:42:47.570327+0000 mon.a (mon.0) 135 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447468] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:15:29 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447468/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447468/ "2023-11-05T01:49:52.688287+0000 mon.a (mon.0) 149 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447473] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:12:39 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447473/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447473/ "2023-11-05T01:49:39.194233+0000 mon.a (mon.0) 229 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447478] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} ----------------------------------------------------------------- time: 02:01:40 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447478/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447478/ "2023-11-05T02:00:06.876992+0000 mon.a (mon.0) 232 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447481] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:29:43 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447481/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447481/ "2023-11-05T02:05:29.691394+0000 mon.a (mon.0) 341 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7447487] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 01:06:52 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447487/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447487/ "2023-11-05T02:12:57.304720+0000 mon.a (mon.0) 205 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log Dead ================================================================= [7447334] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447334/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447334/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7447354] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447354/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447354/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7447391] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447391/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447391/ hit max job timeout Pass ================================================================= [7447296] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:03 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447296/ [7447297] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:18:54 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447297/ [7447298] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:13:42 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447298/ [7447300] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:21 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447300/ [7447302] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:05 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447302/ [7447303] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:28:44 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447303/ [7447304] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:20:53 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447304/ [7447305] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:40:35 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447305/ [7447306] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:51 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447306/ [7447307] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:15:20 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447307/ [7447308] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:34:16 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447308/ [7447309] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:20:07 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447309/ [7447310] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:13:45 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447310/ [7447313] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:27 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447313/ [7447314] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:35:48 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447314/ [7447316] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:47 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447316/ [7447317] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:29:29 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447317/ [7447318] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:50:34 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447318/ [7447320] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:18:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447320/ [7447321] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:28:13 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447321/ [7447322] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:48 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447322/ [7447323] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:35:07 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447323/ [7447324] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:44 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447324/ [7447326] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:42:27 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447326/ [7447327] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:25 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447327/ [7447328] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:22:52 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447328/ [7447329] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:47:55 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447329/ [7447330] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:17 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447330/ [7447331] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:14:40 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447331/ [7447333] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:20 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447333/ [7447335] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:10:59 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447335/ [7447336] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:27:40 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447336/ [7447337] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:16:55 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447337/ [7447338] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:33:53 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447338/ [7447339] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:26 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447339/ [7447341] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:21:47 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447341/ [7447342] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:26:32 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447342/ [7447343] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:20:25 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447343/ [7447345] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:12 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447345/ [7447346] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:15:11 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447346/ [7447347] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:17 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447347/ [7447348] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:07:39 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447348/ [7447349] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:43 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447349/ [7447350] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:21:28 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447350/ [7447352] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:35:19 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447352/ [7447353] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:31 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447353/ [7447355] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:36:16 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447355/ [7447357] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:00:57 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447357/ [7447358] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:17:07 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447358/ [7447359] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:51:16 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447359/ [7447360] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:19:54 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447360/ [7447362] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:19:11 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447362/ [7447364] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:52:50 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447364/ [7447365] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:08:59 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447365/ [7447366] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:26:34 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447366/ [7447367] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} time: 01:00:41 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447367/ [7447369] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:34:28 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447369/ [7447370] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:22:27 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447370/ [7447371] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447371/ [7447372] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:36 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447372/ [7447373] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:16:06 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447373/ [7447375] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:21:04 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447375/ [7447376] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} time: 02:26:13 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447376/ [7447377] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:13:50 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447377/ [7447378] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:15:02 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447378/ [7447379] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:17 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447379/ [7447380] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:24 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447380/ [7447381] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:14:37 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447381/ [7447382] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:29:53 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447382/ [7447385] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:57:45 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447385/ [7447387] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:22:09 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447387/ [7447388] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:17 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447388/ [7447390] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:50:08 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447390/ [7447392] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:24:24 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447392/ [7447394] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:16:09 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447394/ [7447395] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 02:04:33 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447395/ [7447396] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} time: 00:23:14 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447396/ [7447397] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:43:43 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447397/ [7447398] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:35 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447398/ [7447399] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:34:55 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447399/ [7447402] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:43:28 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447402/ [7447403] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:53:41 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447403/ [7447404] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_20.04} tasks/fstop} time: 00:13:57 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447404/ [7447405] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:59 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447405/ [7447406] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:44:26 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447406/ [7447408] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:38 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447408/ [7447409] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:53:28 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447409/ [7447410] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:14:45 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447410/ [7447411] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:40 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447411/ [7447412] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:50:34 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447412/ [7447413] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:37:57 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447413/ [7447414] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:14:13 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447414/ [7447415] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:54:04 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447415/ [7447418] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:09:21 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447418/ [7447419] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:19:56 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447419/ [7447420] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:47:15 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447420/ [7447421] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 01:08:05 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447421/ [7447422] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:53:16 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447422/ [7447423] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:20:53 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447423/ [7447425] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:39 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447425/ [7447427] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:17:04 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447427/ [7447428] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:20:33 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447428/ [7447429] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:55:53 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447429/ [7447430] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:22:09 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447430/ [7447432] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:50:22 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447432/ [7447433] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:52:43 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447433/ [7447434] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:52:58 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447434/ [7447435] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:25:39 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447435/ [7447438] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:26:43 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447438/ [7447440] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:49:25 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447440/ [7447443] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:33:23 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447443/ [7447444] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:15:17 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447444/ [7447445] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:55:01 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447445/ [7447446] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:26:14 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447446/ [7447447] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:21:06 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447447/ [7447449] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:10:48 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447449/ [7447451] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:37:01 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447451/ [7447452] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:16:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447452/ [7447453] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:18:53 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447453/ [7447454] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:43:46 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447454/ [7447455] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:42:32 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447455/ [7447456] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:17:56 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447456/ [7447458] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:21:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447458/ [7447459] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:25:41 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447459/ [7447460] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:22:36 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447460/ [7447462] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:35:00 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447462/ [7447463] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:44:52 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447463/ [7447464] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:52:58 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447464/ [7447465] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:41:14 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447465/ [7447466] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:15 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447466/ [7447467] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 00:54:29 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447467/ [7447469] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:37:21 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447469/ [7447470] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:19:38 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447470/ [7447471] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:42 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447471/ [7447472] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:48:07 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447472/ [7447474] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:40:13 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447474/ [7447475] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:16:27 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447475/ [7447476] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:41 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447476/ [7447477] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:24:10 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447477/ [7447479] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:17:22 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447479/ [7447480] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:40:37 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447480/ [7447482] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:16:36 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447482/ [7447483] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:21 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447483/ [7447484] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:17:56 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447484/ [7447485] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:56:20 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447485/ [7447486] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:19:03 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447486/ [7447488] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:53:31 info: http://pulpito.ceph.com/yuriw-2023-11-04_11:32:36-fs-wip-yuri-testing-2023-11-01-1538-reef-distro-default-smithi/7447488/