2024-02-01T10:41:50.186 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-01T10:41:50.186 INFO:teuthology.results:subset: '50/1024' 2024-02-01T10:41:50.186 INFO:teuthology.results:seed: '1628' 2024-02-01T14:08:31.277 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-01T14:08:31.295 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-01T15:25:38.311 INFO:teuthology.results:Tests finished! gathering results... 2024-02-01T15:25:54.827 INFO:scrape:Found 201 jobs 2024-02-01T15:26:06.638 WARNING:scrape:Misunderstood line: 2024-02-01T15:26:11.890 INFO:scrape:Found 10 distinct failure reasons 2024-02-01T15:26:11.890 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2024-02-01T15:26:11.891 INFO:scrape:1 jobs: ['7541796'] 2024-02-01T15:26:11.891 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}'] 2024-02-01T15:26:11.891 INFO:scrape: 2024-02-01T15:26:11.891 INFO:scrape:Failure: Test failure: test_subvolume_group_rm_when_its_not_empty (tasks.cephfs.test_volumes.TestSubvolumeGroups) 2024-02-01T15:26:11.891 INFO:scrape:2 jobs: ['7541809', '7541933'] 2024-02-01T15:26:11.892 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}'] 2024-02-01T15:26:11.892 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}'] 2024-02-01T15:26:11.892 INFO:scrape: 2024-02-01T15:26:11.892 INFO:scrape:Failure: No module named 'tasks.kclient' 2024-02-01T15:26:11.892 INFO:scrape:5 jobs: ['7541855', '7541861', '7541856', '7541864', '7541862'] 2024-02-01T15:26:11.892 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2024-02-01T15:26:11.893 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'frag', 'fs/functional/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/journal-repair}', 'tasks/volumes/{overrides', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'test/finisher_per_module}}', 'thrashosds-health}'] 2024-02-01T15:26:11.893 INFO:scrape: 2024-02-01T15:26:11.893 INFO:scrape:Failure: Command failed on smithi160 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' 2024-02-01T15:26:11.893 INFO:scrape:1 jobs: ['7541865'] 2024-02-01T15:26:11.893 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-flush}'] 2024-02-01T15:26:11.893 INFO:scrape: 2024-02-01T15:26:11.893 INFO:scrape:Failure: Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) 2024-02-01T15:26:11.894 INFO:scrape:1 jobs: ['7541808'] 2024-02-01T15:26:11.894 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2024-02-01T15:26:11.894 INFO:scrape: 2024-02-01T15:26:11.894 INFO:scrape:Failure: "2024-02-01T14:42:08.498648+0000 mon.a (mon.0) 8238 : cluster [WRN] Health check failed: Degraded data redundancy: 1/50 objects degraded (2.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2024-02-01T15:26:11.894 INFO:scrape:4 jobs: ['7541959', '7541858', '7541801', '7541799'] 2024-02-01T15:26:11.894 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2024-02-01T15:26:11.894 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distros/ubuntu_latest', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/admin}', 'tasks/cephfs_misc_tests}', 'tasks/fragment}', 'tasks/mirror}'] 2024-02-01T15:26:11.894 INFO:scrape: 2024-02-01T15:26:11.895 INFO:scrape:Failure: "2024-02-01T12:26:07.163548+0000 osd.7 (osd.7) 3 : cluster [WRN] OSD bench result of 109194.576436 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2024-02-01T15:26:11.895 INFO:scrape:1 jobs: ['7541873'] 2024-02-01T15:26:11.895 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2024-02-01T15:26:11.895 INFO:scrape: 2024-02-01T15:26:11.895 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6726a3ce665776db7df4de482d481b17c20a4c23 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-02-01T15:26:11.895 INFO:scrape:1 jobs: ['7541934'] 2024-02-01T15:26:11.895 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2024-02-01T15:26:11.896 INFO:scrape: 2024-02-01T15:26:11.896 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2024-02-01T15:26:11.896 INFO:scrape:1 jobs: ['7541914'] 2024-02-01T15:26:11.896 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2024-02-01T15:26:11.896 INFO:scrape: 2024-02-01T15:26:11.896 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-02-01T15:26:11.896 INFO:scrape:1 jobs: ['7541890'] 2024-02-01T15:26:11.897 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-02-01T15:26:11.897 INFO:scrape: 2024-02-01T15:26:11.908 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/ failed: 18 dead: 0 running: 0 waiting: 0 queued: 0 passed: 183 Fail ================================================================= [7541796] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 01:35:38 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541796/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541796/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eaa7e8bd99bd4976805e37cc3fb8757a Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7541799] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} ----------------------------------------------------------------- time: 02:04:39 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541799/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541799/ "2024-02-01T11:30:17.089003+0000 mon.a (mon.0) 2136 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7541801] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:59:45 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541801/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541801/ "2024-02-01T11:48:01.566855+0000 mon.a (mon.0) 1489 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log [7541808] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:25:28 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541808/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541808/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4fc5180754d41df9b12f05d4a581a5d Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) [7541809] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 01:04:52 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541809/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541809/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18a8f8af13504e68b24ecb9fc681659b Test failure: test_subvolume_group_rm_when_its_not_empty (tasks.cephfs.test_volumes.TestSubvolumeGroups) [7541855] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} ----------------------------------------------------------------- time: 00:19:07 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541855/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541855/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b79f58e1de2491b89657e25755c8e41 No module named 'tasks.kclient' [7541856] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:16:22 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541856/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541856/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6eb6f687c7c43ccae7266f1d86ebd7c No module named 'tasks.ceph_fuse' [7541858] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:19:32 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541858/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541858/ "2024-02-01T12:04:21.731958+0000 mon.a (mon.0) 663 : cluster [WRN] Health check failed: Degraded data redundancy: 36/4262 objects degraded (0.845%), 1 pg degraded (PG_DEGRADED)" in cluster log [7541861] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:15:36 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541861/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541861/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dab608d639e247d7864cb09305dd538e No module named 'tasks.mds_thrash' [7541862] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:09:37 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541862/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541862/ sentry: https://sentry.ceph.com/organizations/ceph/?query=91c9bd16c4fd4f9588cbd1fdeacdb246 No module named 'tasks.cephfs.fuse_mount' [7541864] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:12:06 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541864/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66b80d0b610842b3beb1138748986079 No module named 'tasks.ceph' [7541865] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:09:39 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541865/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e98b4b25b9445a593b78c5cbd2c880f Command failed on smithi160 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib' [7541873] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:24:54 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541873/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541873/ "2024-02-01T12:26:07.163548+0000 osd.7 (osd.7) 3 : cluster [WRN] OSD bench result of 109194.576436 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.7. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7541890] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:11:33 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541890/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541890/ error during scrub thrashing: rank damage found: {'backtrace'} [7541914] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:28:02 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541914/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541914/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aa33400699cf4b91bb94c7cf7556aaf4 saw valgrind issues [7541933] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 01:03:06 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541933/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541933/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f009f5444ca94c43b1b87562fbbf0ee7 Test failure: test_subvolume_group_rm_when_its_not_empty (tasks.cephfs.test_volumes.TestSubvolumeGroups) [7541934] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:54:24 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541934/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541934/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a79ea1fb092439fa0e0c55bbf5cb038 Command failed (workunit test suites/dbench.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6726a3ce665776db7df4de482d481b17c20a4c23 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7541959] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 01:27:31 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541959/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541959/ "2024-02-01T14:42:08.498648+0000 mon.a (mon.0) 8238 : cluster [WRN] Health check failed: Degraded data redundancy: 1/50 objects degraded (2.000%), 1 pg degraded (PG_DEGRADED)" in cluster log Pass ================================================================= [7541787] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:26:04 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541787/ [7541788] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:13:13 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541788/ [7541789] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:21:27 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541789/ [7541790] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:41:11 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541790/ [7541791] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:21:00 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541791/ [7541792] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:16:32 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541792/ [7541793] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:13 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541793/ [7541794] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:12:19 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541794/ [7541795] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 01:02:10 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541795/ [7541797] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:24:51 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541797/ [7541798] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:14:49 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541798/ [7541800] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{centos_latest} workloads/cephfs-mirror-ha-workunit} time: 00:36:27 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541800/ [7541802] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:41:06 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541802/ [7541803] fs/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{ubuntu_20.04} tasks/nfs} time: 00:39:26 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541803/ [7541804] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:21:34 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541804/ [7541805] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:48:44 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541805/ [7541806] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:05:45 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541806/ [7541807] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_20.04} tasks/fstop} time: 00:16:31 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541807/ [7541810] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:34:30 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541810/ [7541811] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:27:03 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541811/ [7541812] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:22:01 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541812/ [7541813] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:37:58 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541813/ [7541814] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:14:15 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541814/ [7541815] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:15:42 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541815/ [7541816] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:26:21 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541816/ [7541817] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:14:25 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541817/ [7541818] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:16:59 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541818/ [7541819] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:30:00 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541819/ [7541820] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 00:56:13 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541820/ [7541821] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:13:56 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541821/ [7541822] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:37:01 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541822/ [7541823] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:20:19 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541823/ [7541824] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:22:35 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541824/ [7541825] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 01:29:38 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541825/ [7541826] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 01:16:19 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541826/ [7541827] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:34 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541827/ [7541828] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:00:32 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541828/ [7541829] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:40:51 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541829/ [7541830] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:58:29 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541830/ [7541831] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 01:32:36 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541831/ [7541832] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:35:44 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541832/ [7541833] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 01:19:59 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541833/ [7541834] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:05:30 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541834/ [7541835] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 02:18:39 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541835/ [7541836] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:14:45 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541836/ [7541837] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:51:12 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541837/ [7541838] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:53:39 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541838/ [7541839] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 01:10:16 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541839/ [7541840] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 01:44:55 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541840/ [7541841] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:34:52 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541841/ [7541842] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:27:11 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541842/ [7541843] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:20:38 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541843/ [7541844] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} time: 01:11:41 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541844/ [7541845] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 01:04:03 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541845/ [7541846] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:42:22 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541846/ [7541847] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 01:02:34 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541847/ [7541848] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:56:56 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541848/ [7541849] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:46:33 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541849/ [7541850] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:19:00 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541850/ [7541851] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:30 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541851/ [7541852] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:39:07 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541852/ [7541853] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:18:53 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541853/ [7541854] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:24:14 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541854/ [7541857] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:53:07 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541857/ [7541859] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:27:30 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541859/ [7541860] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:38:50 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541860/ [7541863] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:44:58 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541863/ [7541866] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:32:34 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541866/ [7541867] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:28:36 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541867/ [7541868] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:44:28 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541868/ [7541869] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:47:46 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541869/ [7541870] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:45:50 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541870/ [7541871] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:42 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541871/ [7541872] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:37:58 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541872/ [7541874] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:18 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541874/ [7541875] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:18:33 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541875/ [7541876] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:30 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541876/ [7541877] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:43 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541877/ [7541878] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:13:49 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541878/ [7541879] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:46:16 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541879/ [7541880] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} time: 01:54:52 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541880/ [7541881] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:23:40 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541881/ [7541882] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:23:40 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541882/ [7541883] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:33:39 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541883/ [7541884] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 01:16:46 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541884/ [7541885] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:21:04 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541885/ [7541886] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:23:43 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541886/ [7541887] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:49:25 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541887/ [7541888] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:14:03 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541888/ [7541889] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:52:13 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541889/ [7541891] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:18:29 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541891/ [7541892] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:03 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541892/ [7541893] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:13:43 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541893/ [7541894] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:17:09 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541894/ [7541895] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:20:45 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541895/ [7541896] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:27 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541896/ [7541897] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:23:43 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541897/ [7541898] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:31:43 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541898/ [7541899] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:08 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541899/ [7541900] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:53:10 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541900/ [7541901] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:07 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541901/ [7541902] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 02:01:51 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541902/ [7541903] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} time: 00:41:04 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541903/ [7541904] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:30:48 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541904/ [7541905] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:50 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541905/ [7541906] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:18:07 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541906/ [7541907] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:33:10 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541907/ [7541908] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:47:28 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541908/ [7541909] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:34:37 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541909/ [7541910] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:13:30 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541910/ [7541911] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:29:26 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541911/ [7541912] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 00:53:19 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541912/ [7541913] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 01:00:49 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541913/ [7541915] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:15:08 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541915/ [7541916] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:17:45 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541916/ [7541917] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:52:10 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541917/ [7541918] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} time: 02:18:31 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541918/ [7541919] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:29:42 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541919/ [7541920] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:19:07 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541920/ [7541921] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:41:54 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541921/ [7541922] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:15:36 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541922/ [7541923] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:35 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541923/ [7541924] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:18:43 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541924/ [7541925] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:10 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541925/ [7541926] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:19:47 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541926/ [7541927] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:31:18 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541927/ [7541928] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:19:47 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541928/ [7541929] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:38:52 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541929/ [7541930] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:14:35 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541930/ [7541931] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:23 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541931/ [7541932] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:39:30 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541932/ [7541935] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:42 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541935/ [7541936] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:52:39 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541936/ [7541937] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:23:36 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541937/ [7541938] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:23:50 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541938/ [7541939] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:14:09 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541939/ [7541940] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:51:10 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541940/ [7541941] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:19:58 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541941/ [7541942] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:02 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541942/ [7541943] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:49:12 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541943/ [7541944] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:16:15 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541944/ [7541945] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:16:14 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541945/ [7541946] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:14 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541946/ [7541947] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:13:07 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541947/ [7541948] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:34 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541948/ [7541949] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:24:14 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541949/ [7541950] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:16:18 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541950/ [7541951] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:22:42 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541951/ [7541952] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:20:46 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541952/ [7541953] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:40:13 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541953/ [7541954] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:30:58 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541954/ [7541955] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:17:47 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541955/ [7541956] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:50:22 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541956/ [7541957] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:18:50 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541957/ [7541958] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:12:06 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541958/ [7541960] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:46 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541960/ [7541961] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:20:06 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541961/ [7541962] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:28:23 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541962/ [7541963] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:15:29 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541963/ [7541964] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:18:08 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541964/ [7541965] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:52:07 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541965/ [7541966] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:57:26 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541966/ [7541967] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:19:10 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541967/ [7541968] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:29:05 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541968/ [7541969] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:25:01 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541969/ [7541970] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:32 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541970/ [7541971] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:17:19 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541971/ [7541972] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:19:08 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541972/ [7541973] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:39:36 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541973/ [7541974] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:05 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541974/ [7541975] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:46:27 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541975/ [7541976] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:22:01 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541976/ [7541977] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:15:30 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541977/ [7541978] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 00:58:02 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541978/ [7541979] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:36 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541979/ [7541980] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:12:16 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541980/ [7541981] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:15:08 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541981/ [7541982] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:26:46 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541982/ [7541983] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:20:43 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541983/ [7541984] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:31:08 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541984/ [7541985] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:27:10 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541985/ [7541986] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:26:31 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541986/ [7541987] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:56:39 info: https://pulpito.ceph.com/vshankar-2024-02-01_07:00:26-fs-wip-vshankar-testing1-reef-2024-01-29-1551-reef-testing-default-smithi/7541987/