2026-02-23T08:54:12.377 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-23T08:54:12.377 INFO:teuthology.results:subset: '111/120000'
2026-02-23T08:54:12.377 INFO:teuthology.results:seed: '1337'
2026-02-23T09:37:47.467 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-23T09:37:47.473 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2026-02-23T15:37:26.682 INFO:teuthology.results:Tests finished! gathering results...
2026-02-23T15:37:29.094 INFO:scrape:Found 93 jobs
2026-02-23T15:37:34.473 WARNING:scrape:Misunderstood line: 
2026-02-23T15:37:37.145 WARNING:scrape:Misunderstood line: 
2026-02-23T15:38:00.556 INFO:scrape:Found 23 distinct failure reasons
2026-02-23T15:38:00.556 INFO:scrape:Failure: failed to install new kernel version within timeout
2026-02-23T15:38:00.556 INFO:scrape:21 jobs: ['65966', '65947', '65928', '65943', '65882', '65905', '65909', '65939', '65948', '65967', '65937', '65894', '65915', '65961', '65952', '65919', '65896', '65958', '65891', '65911', '65884']
2026-02-23T15:38:00.556 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}']
2026-02-23T15:38:00.557 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/iozone}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', '3-modules}', 'clusters/1-mds-1-client-coloc', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/ubuntu_20.04', 'distro/{ubuntu/{latest', 'distro/{ubuntu_20.04}', 'frag', 'fs/32bits/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/shell/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'overrides}}', 'pg_health', 'pg_health}', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'prefetch_oft_dirfrags/no', 'prefetch_oft_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'session_timeout', 'session_timeout}', 'tasks/asok_dump_tree}', 'tasks/cephfs-shell}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/dbench', 'tasks/fsstress', 'tasks/test_journal_migration}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/basic}}', 'test/clone}}', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}']
2026-02-23T15:38:00.557 INFO:scrape:
2026-02-23T15:38:00.557 INFO:scrape:Failure: 
2026-02-23T15:38:00.557 INFO:scrape:14 jobs: ['65898', '65879', '65929', '65942', '65889', '65938', '65959', '65899', '65888', '65901', '65892', '65890', '65907', '65969']
2026-02-23T15:38:00.557 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}']
2026-02-23T15:38:00.557 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-client', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/fuse', '2-logrotate', '2-scrub/no', '2-upgrade', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/suites/fsstress}}', '3-client-upgrade', '3-inline/no', '3-modules}', '3-snaps/no', '3-upgrade-mgr-staggered', '4-compat_client', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-client-sanity}}', '5-upgrade-with-workload', '5-workunit/suites/fsstress}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_9.stream}', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'export-check', 'fail_fs/yes', 'frag', 'fs/32bits/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'msgr-failures/none', 'multifs', 'multimds/no', 'multimds/yes', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'prefetch_oft_dirfrags/no', 'prefetch_oft_dirfrags/yes', 'races', 'ranks/5', 'ranks/multi/{balancer/automatic', 'replication/always}', 'roles', 'session_timeout', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/fsstress', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'traceless/50pc}', 'ubuntu_22.04}', 'validater/lockdep}', 'wsync/no}']
2026-02-23T15:38:00.557 INFO:scrape:
2026-02-23T15:38:00.557 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/kernel_untar_build.sh
2026-02-23T15:38:00.557 INFO:scrape:3 jobs: ['65936', '65968', '65880']
2026-02-23T15:38:00.557 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mgr', 'mon', 'ms-die-on-skipped}', 'osd-asserts', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'pg_health}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2026-02-23T15:38:00.557 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mgr', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'pg_health}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2026-02-23T15:38:00.557 INFO:scrape:
2026-02-23T15:38:00.557 INFO:scrape:Failure: Test failure: test_filelock (tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock)
2026-02-23T15:38:00.557 INFO:scrape:1 jobs: ['65926']
2026-02-23T15:38:00.557 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/client-recovery}']
2026-02-23T15:38:00.558 INFO:scrape:
2026-02-23T15:38:00.558 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on trial023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2026-02-23T15:38:00.558 INFO:scrape:1 jobs: ['65953']
2026-02-23T15:38:00.558 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}']
2026-02-23T15:38:00.558 INFO:scrape:
2026-02-23T15:38:00.558 INFO:scrape:Failure: Command failed (workunit test fs/misc/fallocate.sh) on trial031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/fallocate.sh'
2026-02-23T15:38:00.558 INFO:scrape:1 jobs: ['65935']
2026-02-23T15:38:00.558 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}']
2026-02-23T15:38:00.558 INFO:scrape:
2026-02-23T15:38:00.558 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=pacific
2026-02-23T15:38:00.558 INFO:scrape:4 jobs: ['65946', '65881', '65914', '65904']
2026-02-23T15:38:00.558 INFO:scrape:suites intersection: ['1-volume/{0-create', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}']
2026-02-23T15:38:00.558 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4']
2026-02-23T15:38:00.558 INFO:scrape:
2026-02-23T15:38:00.558 INFO:scrape:Failure: "grep: /var/log/ceph/9ef5fb60-1099-11f1-ad26-d404e6e7d460/ceph.log: No such file or directory" in cluster log
2026-02-23T15:38:00.558 INFO:scrape:3 jobs: ['65931', '65954', '65887']
2026-02-23T15:38:00.558 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2026-02-23T15:38:00.558 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2026-02-23T15:38:00.558 INFO:scrape:
2026-02-23T15:38:00.558 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFSCryptXFS.test_fscrypt_dummy_encryption_with_quick_group)
2026-02-23T15:38:00.559 INFO:scrape:1 jobs: ['65885']
2026-02-23T15:38:00.559 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate', '3-modules}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-client']
2026-02-23T15:38:00.559 INFO:scrape:
2026-02-23T15:38:00.559 INFO:scrape:Failure: Test failure: test_subvolume_replicated (tasks.cephfs.test_subvolume.TestSubvolumeReplicated.test_subvolume_replicated)
2026-02-23T15:38:00.559 INFO:scrape:1 jobs: ['65897']
2026-02-23T15:38:00.559 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/volumes/{overrides', 'test/basic}}']
2026-02-23T15:38:00.559 INFO:scrape:
2026-02-23T15:38:00.559 INFO:scrape:Dead: b'2026-02-23T15:19:12.050 DEBUG:teuthology.exit:Finished running handlers'
2026-02-23T15:38:00.559 INFO:scrape:9 jobs: ['65934', '65878', '65913', '65964', '65883', '65886', '65903', '65924', '65950']
2026-02-23T15:38:00.559 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}']
2026-02-23T15:38:00.559 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', '3-modules}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'frag', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'prefetch_oft_dirfrags/no', 'prefetch_oft_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'ubuntu_22.04}']
2026-02-23T15:38:00.559 INFO:scrape:
2026-02-23T15:38:00.559 INFO:scrape:Failure: Test failure: test_with_health_warn_trim (tasks.cephfs.test_admin.TestFSFail.test_with_health_warn_trim)
2026-02-23T15:38:00.559 INFO:scrape:1 jobs: ['65906']
2026-02-23T15:38:00.559 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/admin}']
2026-02-23T15:38:00.559 INFO:scrape:
2026-02-23T15:38:00.559 INFO:scrape:Valgrind: client (Leak_StillReachable), mon (Leak_StillReachable)
2026-02-23T15:38:00.559 INFO:scrape:2 jobs: ['65925', '65955']
2026-02-23T15:38:00.559 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_9.stream}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2026-02-23T15:38:00.560 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_9.stream}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2026-02-23T15:38:00.560 INFO:scrape:
2026-02-23T15:38:00.560 INFO:scrape:Failure: Test failure: test_cephfs_mirror_incremental_sync_with_type_mixup (tasks.cephfs.test_mirroring.TestMirroring.test_cephfs_mirror_incremental_sync_with_type_mixup)
2026-02-23T15:38:00.560 INFO:scrape:1 jobs: ['65922']
2026-02-23T15:38:00.560 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/mirror}}']
2026-02-23T15:38:00.560 INFO:scrape:
2026-02-23T15:38:00.560 INFO:scrape:Timeout 5h running clone.client.0/qa/workunits/kernel_untar_build.sh
2026-02-23T15:38:00.560 INFO:scrape:1 jobs: ['65940']
2026-02-23T15:38:00.560 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/no', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'ubuntu_22.04}', 'wsync/yes}']
2026-02-23T15:38:00.560 INFO:scrape:
2026-02-23T15:38:00.560 INFO:scrape:Failure: Test failure: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir)
2026-02-23T15:38:00.560 INFO:scrape:2 jobs: ['65877', '65956']
2026-02-23T15:38:00.560 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no_client_pidfile', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/scrub}']
2026-02-23T15:38:00.560 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/scrub}', 'ubuntu_22.04}']
2026-02-23T15:38:00.560 INFO:scrape:
2026-02-23T15:38:00.560 INFO:scrape:Failure: error during thrashing
2026-02-23T15:38:00.560 INFO:scrape:4 jobs: ['65945', '65965', '65900', '65933']
2026-02-23T15:38:00.560 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'conf/{client', 'frag', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{client-shutdown', 'pg_health', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2026-02-23T15:38:00.560 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/pjd}}', '3-modules}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'pg_health', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'prefetch_oft_dirfrags/no', 'prefetch_oft_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'ubuntu_22.04}']
2026-02-23T15:38:00.560 INFO:scrape:
2026-02-23T15:38:00.560 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs.test_acls)
2026-02-23T15:38:00.560 INFO:scrape:1 jobs: ['65902']
2026-02-23T15:38:00.561 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/acls}']
2026-02-23T15:38:00.561 INFO:scrape:
2026-02-23T15:38:00.561 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on trial140 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench'
2026-02-23T15:38:00.561 INFO:scrape:1 jobs: ['65960']
2026-02-23T15:38:00.561 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-dbench}}', '2-logrotate', '3-modules}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-client']
2026-02-23T15:38:00.561 INFO:scrape:
2026-02-23T15:38:00.561 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_clone_pending_for_no_group (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_snapshot_info_if_clone_pending_for_no_group)
2026-02-23T15:38:00.561 INFO:scrape:1 jobs: ['65908']
2026-02-23T15:38:00.561 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/volumes/{overrides', 'test/clone}}', 'ubuntu_22.04}']
2026-02-23T15:38:00.561 INFO:scrape:
2026-02-23T15:38:00.561 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_sync (tasks.cephfs.test_mirroring.TestMirroring.test_cephfs_mirror_cancel_sync)
2026-02-23T15:38:00.561 INFO:scrape:1 jobs: ['65916']
2026-02-23T15:38:00.561 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'conf/{client', 'fs/mirror/{begin/{0-install', 'mds', 'mgr', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/mirror}']
2026-02-23T15:38:00.561 INFO:scrape:
2026-02-23T15:38:00.561 INFO:scrape:Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS.test_cluster_set_reset_user_config)
2026-02-23T15:38:00.561 INFO:scrape:1 jobs: ['65918']
2026-02-23T15:38:00.561 INFO:scrape:suites: ['conf/{client', 'fs/nfs/{cluster/{1-node}', 'ignorelist_health', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignore_mgr_down', 'pg_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/nfs}']
2026-02-23T15:38:00.561 INFO:scrape:
2026-02-23T15:38:00.561 INFO:scrape:Failure: Test failure: test_client_evict_with_invalid_id (tasks.cephfs.test_misc.TestSessionClientEvict.test_client_evict_with_invalid_id)
2026-02-23T15:38:00.561 INFO:scrape:2 jobs: ['65917', '65963']
2026-02-23T15:38:00.561 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cephfs_misc_tests}']
2026-02-23T15:38:00.561 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cephfs_misc_tests}']
2026-02-23T15:38:00.562 INFO:scrape:
2026-02-23T15:38:00.564 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial
=================================================================
info:    https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/
logs:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/
failed:  68
dead:    9
running: 0
waiting: 0
queued:  0
passed:  16


Fail
=================================================================
[65877]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/scrub}
-----------------------------------------------------------------
time:   00:16:53
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65877/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65877/

    Test failure: test_flag_scrub_mdsdir
    (tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir)

[65879]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   02:27:17
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65879/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65879/

[65880]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   03:15:42
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65880/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65880/

    Command failed (workunit test kernel_untar_build.sh) on trial091 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[65881]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:03:53
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65881/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65881/

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=paci
    fic

[65882]  fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/3 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65882/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65882/

    failed to install new kernel version within timeout

[65884]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65884/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65884/

    failed to install new kernel version within timeout

[65885]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-common}}
-----------------------------------------------------------------
time:   00:11:35
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65885/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65885/

    Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.
    test_fscrypt.TestFSCryptXFS.test_fscrypt_dummy_encryption_with_quick_group)

[65887]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:05:48
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65887/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65887/

    "grep: /var/log/ceph/ce112d40-1097-11f1-8bca-d404e6e7d460/ceph.log: No such
    file or directory" in cluster log

[65888]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   01:53:13
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65888/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65888/

[65889]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/quincy 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   02:32:03
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65889/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65889/

[65890]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   02:27:30
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65890/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65890/

[65891]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65891/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65891/

    failed to install new kernel version within timeout

[65892]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   02:38:50
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65892/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65892/

[65894]  fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65894/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65894/

    failed to install new kernel version within timeout

[65896]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65896/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65896/

    failed to install new kernel version within timeout

[65897]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   02:10:04
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65897/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65897/

    Test failure: test_subvolume_replicated (tasks.cephfs.test_subvolume.TestSu
    bvolumeReplicated.test_subvolume_replicated)

[65898]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   02:07:13
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65898/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65898/

[65899]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
-----------------------------------------------------------------
time:   02:22:07
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65899/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65899/

[65900]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:09:13
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65900/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65900/

    error during thrashing

[65901]  fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_9.stream} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   02:08:21
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65901/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65901/

[65902]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/acls}
-----------------------------------------------------------------
time:   00:10:15
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65902/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65902/

    Test failure: test_acls (tasks.cephfs.test_acls.TestACLs.test_acls)

[65904]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:03:48
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65904/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65904/

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3cb
    e25cde3cfa028984618ad32de9edc4c1eaed0

[65905]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65905/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65905/

    failed to install new kernel version within timeout

[65906]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/admin}
-----------------------------------------------------------------
time:   00:27:11
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65906/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65906/

    Test failure: test_with_health_warn_trim
    (tasks.cephfs.test_admin.TestFSFail.test_with_health_warn_trim)

[65907]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   02:28:26
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65907/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65907/

[65908]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:49:08
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65908/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65908/

    Test failure: test_subvolume_snapshot_info_if_clone_pending_for_no_group (t
    asks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_subvolume_snapsho
    t_info_if_clone_pending_for_no_group)

[65909]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/asok_dump_tree}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65909/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65909/

    failed to install new kernel version within timeout

[65911]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65911/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65911/

    failed to install new kernel version within timeout

[65914]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:03:58
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65914/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65914/

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=paci
    fic

[65915]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65915/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65915/

    failed to install new kernel version within timeout

[65916]  fs/mirror/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} conf/{client mds mgr mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/mirror}
-----------------------------------------------------------------
time:   00:38:30
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65916/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65916/

    Test failure: test_cephfs_mirror_cancel_sync
    (tasks.cephfs.test_mirroring.TestMirroring.test_cephfs_mirror_cancel_sync)

[65917]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:51:53
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65917/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65917/

    Test failure: test_client_evict_with_invalid_id (tasks.cephfs.test_misc.Tes
    tSessionClientEvict.test_client_evict_with_invalid_id)

[65918]  fs/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignore_mgr_down ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs}
-----------------------------------------------------------------
time:   00:12:38
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65918/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65918/

    Test failure: test_cluster_set_reset_user_config
    (tasks.cephfs.test_nfs.TestNFS.test_cluster_set_reset_user_config)

[65919]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65919/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65919/

    failed to install new kernel version within timeout

[65922]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/{ignorelist_health pg_health} tasks/mirror}}
-----------------------------------------------------------------
time:   01:10:29
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65922/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65922/

    Test failure: test_cephfs_mirror_incremental_sync_with_type_mixup (tasks.ce
    phfs.test_mirroring.TestMirroring.test_cephfs_mirror_incremental_sync_with_
    type_mixup)

[65925]  fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_9.stream} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:41:22
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65925/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65925/

    saw valgrind issues

[65926]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:30:51
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65926/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65926/

    Test failure: test_filelock
    (tasks.cephfs.test_client_recovery.TestClientRecovery.test_filelock)

[65928]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65928/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65928/

    failed to install new kernel version within timeout

[65929]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   02:26:44
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65929/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65929/

[65931]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:05:27
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65931/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65931/

    "grep: /var/log/ceph/9ef5fb60-1099-11f1-ad26-d404e6e7d460/ceph.log: No such
    file or directory" in cluster log

[65933]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
-----------------------------------------------------------------
time:   00:34:24
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65933/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65933/

    error during thrashing

[65935]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:25:05
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65935/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65935/

    Command failed (workunit test fs/misc/fallocate.sh) on trial031 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/fallocate.sh'

[65936]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   03:17:18
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65936/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65936/

    Command failed (workunit test kernel_untar_build.sh) on trial192 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[65937]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65937/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65937/

    failed to install new kernel version within timeout

[65938]  fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/quincy 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
-----------------------------------------------------------------
time:   02:35:45
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65938/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65938/

[65939]  fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/3 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65939/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65939/

    failed to install new kernel version within timeout

[65940]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}}
-----------------------------------------------------------------
time:   05:21:38
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65940/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65940/

    Command failed (workunit test kernel_untar_build.sh) on trial069 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 5h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'

[65942]  fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   02:25:56
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65942/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65942/

[65943]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65943/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65943/

    failed to install new kernel version within timeout

[65945]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:32:37
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65945/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65945/

    error during thrashing

[65946]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:04:43
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65946/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65946/

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=paci
    fic

[65947]  fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_20.04 mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65947/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65947/

    failed to install new kernel version within timeout

[65948]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65948/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65948/

    failed to install new kernel version within timeout

[65952]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65952/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65952/

    failed to install new kernel version within timeout

[65953]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:28:13
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65953/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65953/

    Command failed (workunit test suites/dbench.sh) on trial023 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[65954]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:05:11
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65954/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65954/

    "grep: /var/log/ceph/cf9c973b-109a-11f1-9ed1-d404e6e7d460/ceph.log: No such
    file or directory" in cluster log

[65955]  fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_9.stream} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:40:59
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65955/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65955/

    saw valgrind issues

[65956]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/scrub}
-----------------------------------------------------------------
time:   00:14:27
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65956/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65956/

    Test failure: test_flag_scrub_mdsdir
    (tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir)

[65958]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65958/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65958/

    failed to install new kernel version within timeout

[65959]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   02:29:02
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65959/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65959/

[65960]  fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-dbench}}
-----------------------------------------------------------------
time:   00:08:40
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65960/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65960/

    Command failed (workunit test fs/fscrypt.sh) on trial140 with status 128:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none
    dbench'

[65961]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65961/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65961/

    failed to install new kernel version within timeout

[65963]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:51:54
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65963/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65963/

    Test failure: test_client_evict_with_invalid_id (tasks.cephfs.test_misc.Tes
    tSessionClientEvict.test_client_evict_with_invalid_id)

[65965]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:06:20
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65965/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65965/

    error during thrashing

[65966]  fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65966/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65966/

    failed to install new kernel version within timeout

[65967]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65967/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65967/

    failed to install new kernel version within timeout

[65968]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   03:17:21
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65968/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65968/

    Command failed (workunit test kernel_untar_build.sh) on trial103 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=b57d1d325914bb542eeaf580bbf1572fbf73cb3d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[65969]  fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
-----------------------------------------------------------------
time:   02:23:45
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65969/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65969/



Dead
=================================================================
[65878]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65878/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65878/

    hit max job timeout

[65883]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65883/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65883/

    hit max job timeout

[65886]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65886/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65886/

    hit max job timeout

[65903]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65903/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65903/

    hit max job timeout

[65913]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65913/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65913/

    hit max job timeout

[65924]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65924/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65924/

    hit max job timeout

[65934]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65934/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65934/

    hit max job timeout

[65950]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65950/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65950/

    hit max job timeout

[65964]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65964/
log:    https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65964/

    killed



Pass
=================================================================
[65893] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd}
time:   00:32:48
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65893/

[65895] fs/upgrade/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/quincy 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}}
time:   00:21:36
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65895/

[65910] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}}
time:   00:26:50
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65910/

[65912] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cap-flush}
time:   00:07:42
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65912/

[65920] fs/top/{begin/{0-install 1-ceph 2-logrotate 3-modules} cluster/{1-node} conf/{client mds mgr mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/fstop}
time:   00:08:25
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65920/

[65921] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:24:02
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65921/

[65923] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/finisher_per_module}}
time:   00:07:50
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65923/

[65927] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}}
time:   00:31:12
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65927/

[65930] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}}
time:   00:18:00
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65930/

[65932] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd}
time:   00:33:46
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65932/

[65941] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock ubuntu_22.04} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/mds-full}
time:   00:14:40
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65941/

[65944] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
time:   00:30:48
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65944/

[65949] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}}
time:   00:28:27
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65949/

[65951] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd}
time:   00:34:05
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65951/

[65957] fs/upgrade/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/quincy 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}}
time:   00:18:17
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65957/

[65962] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs/{frag test}}
time:   01:21:55
info:   https://pulpito.ceph.com/vshankar-2026-02-23_07:56:09-fs-reef-release-testing-default-trial/65962/
2026-02-23T15:38:00.574 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 33, in main
    results(args['--archive-dir'], args['--name'], args['--email'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 78, in results
    email_results(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 95, in email_results
    smtp = smtplib.SMTP('localhost')
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 341, in connect
    self.sock = self._get_socket(host, port, self.timeout)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket
    return socket.create_connection((host, port), timeout,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection
    sock.connect(sa)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect
    self._internal_connect(address)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect
    raise _SocketError(err, strerror(err))
ConnectionRefusedError: [Errno 111] Connection refused