2026-02-19T15:23:43.311 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-19T15:23:43.311 INFO:teuthology.results:subset: '1007/1024' 2026-02-19T15:23:43.311 INFO:teuthology.results:seed: '5944' 2026-02-19T22:02:48.303 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-19T22:02:48.310 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-20T03:59:12.786 INFO:teuthology.results:Tests finished! gathering results... 2026-02-20T03:59:17.974 INFO:scrape:Found 201 jobs 2026-02-20T04:00:02.298 INFO:scrape:Found 32 distinct failure reasons 2026-02-20T04:00:02.298 INFO:scrape:Failure: Failed to reconnect to trial199 2026-02-20T04:00:02.298 INFO:scrape:39 jobs 2026-02-20T04:00:02.298 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}'] 2026-02-20T04:00:02.298 INFO:scrape:suites union: ['0-subvolume/{with-namespace-isolated-and-quota}', '0-subvolume/{with-namespace-isolated}', '0-subvolume/{with-no-extra-options}', '0-subvolume/{with-quota}', '0-use-snaprealm/{global}', '0-use-snaprealm/{subvol}', '1-ceph', '1-cephadm', '1-check-counter', '1-mount/mount/kclient/{mount-syntax/{v1}', '2-clients/kclient', '2-logrotate', '2-scrub/yes', '2-workunit/iozone}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/iozone}}', '3-kernel}', '3-snaps/no', '3-snaps/yes', '3-workload/stress_tests/blogbench}}', '3-workload/stress_tests/fsstress}}', '4-flush/no', '4-flush/yes', '5-quiesce/with-quiesce', '6-workunit/direct_io}}', '6-workunit/fs/misc}}', '6-workunit/fs/norstats}}', '6-workunit/fs/test_o_trunc}}', '6-workunit/kernel_untar_build}}', '6-workunit/postgres}}', '6-workunit/suites/blogbench}}', '6-workunit/suites/ffsb}}', '6-workunit/suites/fsstress}}', '6-workunit/suites/fsync-tester}}', '6-workunit/suites/iogen}}', '6-workunit/suites/iozone}}', '6-workunit/suites/pjd}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro', 'distro/{rpm_latest}', 'export-check', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{0-use-snaprealm/{global}', 'fs/functional/{0-use-snaprealm/{subvol}', 'fs/mixed-clients/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/snaps/{0-use-snaprealm/{subvol}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'kernel', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'n/3', 'n/5', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'osd', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'prefetch_oft_dirfrags/no', 'prefetch_oft_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/admin}', 'tasks/alternate-pool}', 'tasks/auto-repair}', 'tasks/backtrace}', 'tasks/client-limits}', 'tasks/damage}', 'tasks/failover}', 'tasks/forward-scrub}', 'tasks/fscrypt-kbuild}', 'tasks/fsstress', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/multifs-auth}', 'tasks/recovery-fs}', 'tasks/truncate_delay}', 'tasks/workunit/snaps}', 'tasks/xfstests-dev}', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'tasks/{0-install/squid', 'tasks/{1-tests/fscrypt-common}}', 'tasks/{1-thrash/exports', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/with-quiesce', 'thrashosds-health}', 'validater/valgrind}', 'wsync/no}'] 2026-02-20T04:00:02.298 INFO:scrape: 2026-02-20T04:00:02.298 INFO:scrape:Failure: "2026-02-19T21:56:21.498149+0000 mon.a (mon.0) 411 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log 2026-02-20T04:00:02.299 INFO:scrape:42 jobs 2026-02-20T04:00:02.299 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'pg_health', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}'] 2026-02-20T04:00:02.299 INFO:scrape:suites union: ['0-subvolume/{with-namespace-isolated-and-quota}', '0-subvolume/{with-namespace-isolated}', '0-subvolume/{with-no-extra-options}', '0-subvolume/{with-quota}', '0-use-snaprealm/{global}', '0-use-snaprealm/{subvol}', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '2-scrub/yes', '3-kernel}', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-quiesce/no', '5-quiesce/with-quiesce', '6-workunit/direct_io}}', '6-workunit/fs/misc}}', '6-workunit/fs/norstats}}', '6-workunit/fs/test_o_trunc}}', '6-workunit/kernel_untar_build}}', '6-workunit/postgres}}', '6-workunit/suites/blogbench}}', '6-workunit/suites/dbench}}', '6-workunit/suites/ffsb}}', '6-workunit/suites/fsstress}}', '6-workunit/suites/fsx}}', '6-workunit/suites/fsync-tester}}', '6-workunit/suites/iogen}}', '6-workunit/suites/iozone}}', '6-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'pg_health', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'rpm_latest}', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'wsync/yes}'] 2026-02-20T04:00:02.299 INFO:scrape: 2026-02-20T04:00:02.299 INFO:scrape:Failure: valgrind error: MismatchedFree operator delete[](void*, unsigned long, std::align_val_t) UnknownInlinedFun UnknownInlinedFun 2026-02-20T04:00:02.299 INFO:scrape:1 jobs: ['58538'] 2026-02-20T04:00:02.299 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2026-02-20T04:00:02.299 INFO:scrape: 2026-02-20T04:00:02.299 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squid 2026-02-20T04:00:02.299 INFO:scrape:32 jobs 2026-02-20T04:00:02.299 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2026-02-20T04:00:02.299 INFO:scrape:suites union: ['1-client', '1-mount/fuse', '1-mount/kclient/{mount-syntax/{v1}', '1-mount/mount/fuse', '1-ranks/1', '1-ranks/2', '1-upgrade}}', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '2-clients/fuse-upgrade', '2-test_admin_squid_client}}', '2-upgrade', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/yes}}', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '3-workload/stress_tests/dbench}}', '3-workload/stress_tests/fsstress}}', '3-workload/stress_tests/iozone}}', '4-compat_client', '4-config-upgrade/{fail_fs}', '4-verify}', '5-client-sanity}}', '5-upgrade-with-workload', '6-verify}}', 'clusters/1-mds-2-client-micro', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro', 'distro/{rpm_latest}', 'fail_fs/no', 'fail_fs/yes', 'fs/bugs/multifs_mdsauthcaps/{clusters/1node-cluster-1node-client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/upgrade/nofs/{bluestore-bitmap', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'multimds/no', 'multimds/yes', 'no-mds-cluster', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'roles', 'rpm_latest}', 'syntax}', 'tasks/{0-from/reef', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/squid', 'tasks/{0-install', 'tasks/{0-install/reef', 'tasks/{0-install/squid'] 2026-02-20T04:00:02.299 INFO:scrape: 2026-02-20T04:00:02.299 INFO:scrape:Dead: b'2026-02-20T03:39:28.195 DEBUG:teuthology.exit:Finished running handlers' 2026-02-20T04:00:02.300 INFO:scrape:6 jobs: ['58470', '58398', '58466', '58478', '58522', '58360'] 2026-02-20T04:00:02.300 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-kernel}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg_health}'] 2026-02-20T04:00:02.300 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{0-use-snaprealm/{global}', 'fs/functional/{0-use-snaprealm/{subvol}', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'rpm_latest}', 'tasks/admin}', 'tasks/client-recovery}', 'tasks/quiesce}', 'tasks/quota}', 'tasks/scrub}', 'tasks/{1-tests/fscrypt-common}}'] 2026-02-20T04:00:02.300 INFO:scrape: 2026-02-20T04:00:02.300 INFO:scrape:Failure: Command failed on trial007 with status 4: 'rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.17.0_gd74d6c0e9895-2.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://chacra.ceph.com/r/kernel/testing/d74d6c0e98958aa0bdb6f0a93258a856bda58b97/rocky/10/flavors/default/x86_64/ --input-file=-' 2026-02-20T04:00:02.300 INFO:scrape:3 jobs: ['58494', '58449', '58507'] 2026-02-20T04:00:02.300 INFO:scrape:suites intersection: ['2-logrotate', '3-kernel}', 'conf/{client', 'frag', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'pg_health'] 2026-02-20T04:00:02.300 INFO:scrape:suites union: ['0-subvolume/{with-namespace-isolated-and-quota}', '0-subvolume/{with-namespace-isolated}', '0-use-snaprealm/{global}', '1-ceph', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '2-workunit/cfuse_workunit_trivial_sync}}', '3-kernel}', '3-snaps/no', '4-flush/yes', '5-quiesce/with-quiesce', '6-workunit/suites/dbench}}', '6-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rpm_latest}', 'export-check', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'msgr-failures/osd-mds-delay', 'multifs', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'wsync/no}'] 2026-02-20T04:00:02.300 INFO:scrape: 2026-02-20T04:00:02.300 INFO:scrape:Failure: Command 'git ls-remote https://github.com/ceph/ceph-ci v18.2.1^{}' returned non-zero exit status 128. 2026-02-20T04:00:02.300 INFO:scrape:1 jobs: ['58436'] 2026-02-20T04:00:02.300 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'conf/{client', 'distro', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.1}'] 2026-02-20T04:00:02.300 INFO:scrape: 2026-02-20T04:00:02.300 INFO:scrape:Failure: Command failed on trial190 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-osd ceph-osd-classic ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' 2026-02-20T04:00:02.300 INFO:scrape:2 jobs: ['58497', '58413'] 2026-02-20T04:00:02.300 INFO:scrape:suites intersection: ['2-logrotate', '3-kernel}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'osd}'] 2026-02-20T04:00:02.300 INFO:scrape:suites union: ['0-subvolume/{with-quota}', '0-use-snaprealm/{subvol}', '1-ceph', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-kernel}', '3-snaps/yes', '4-flush/no', '5-quiesce/with-quiesce', '6-workunit/suites/fsx}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{rpm_latest}', 'export-check', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'tasks/{1-tests/fscrypt-pjd}}'] 2026-02-20T04:00:02.300 INFO:scrape: 2026-02-20T04:00:02.300 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt_kclient_fuse.sh) on trial064 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/fscrypt-fuse && cd -- /home/ubuntu/cephtest/mnt.0/fscrypt-fuse && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 4h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt_kclient_fuse.sh fuse' 2026-02-20T04:00:02.300 INFO:scrape:1 jobs: ['58352'] 2026-02-20T04:00:02.300 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mgr', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd-asserts', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'rpm_latest}', 'tasks/fscrypt-kbuild}'] 2026-02-20T04:00:02.300 INFO:scrape: 2026-02-20T04:00:02.300 INFO:scrape:Failure: Command failed (workunit test suites/blogbench.sh) on trial135 with status 4: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh' 2026-02-20T04:00:02.300 INFO:scrape:2 jobs: ['58399', '58420'] 2026-02-20T04:00:02.301 INFO:scrape:suites intersection: ['2-logrotate', '3-kernel}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd-asserts', 'osd}', 'rpm_latest}'] 2026-02-20T04:00:02.301 INFO:scrape:suites union: ['0-subvolume/{with-quota}', '0-use-snaprealm/{global}', '1-ceph', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-kernel}', '3-snaps/no', '4-flush/yes', '5-quiesce/no', '6-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rpm_latest}', 'export-check', 'frag', 'fs/mixed-clients/{begin/{0-install', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'ranks/multi/{balancer/automatic', 'replication/always}', 'rpm_latest}', 'session_timeout}', 'standby-replay', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'wsync/yes}'] 2026-02-20T04:00:02.301 INFO:scrape: 2026-02-20T04:00:02.301 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-snap-rm-cmp.sh) on trial055 with status 4: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap-rm-cmp.sh' 2026-02-20T04:00:02.301 INFO:scrape:2 jobs: ['58368', '58461'] 2026-02-20T04:00:02.301 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-kernel}', 'conf/{client', 'distro/{rpm_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}'] 2026-02-20T04:00:02.301 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '2-workunit/fs/snaps}}', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-1c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rpm_latest}', 'frag', 'fs/snaps/{0-use-snaprealm/{subvol}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'prefetch_entire_dirfrags/yes', 'prefetch_oft_dirfrags/yes', 'races', 'ranks/3', 'rpm_latest}', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2026-02-20T04:00:02.301 INFO:scrape: 2026-02-20T04:00:02.301 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration.test_journal_migration) 2026-02-20T04:00:02.301 INFO:scrape:1 jobs: ['58500'] 2026-02-20T04:00:02.301 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/functional/{0-use-snaprealm/{subvol}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/test_journal_migration}'] 2026-02-20T04:00:02.301 INFO:scrape: 2026-02-20T04:00:02.301 INFO:scrape:Failure: Command failed (workunit test suites/iozone.sh) on trial164 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/iozone.sh' 2026-02-20T04:00:02.301 INFO:scrape:3 jobs: ['58480', '58476', '58490'] 2026-02-20T04:00:02.301 INFO:scrape:suites intersection: ['2-logrotate', '3-kernel}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'rpm_latest}'] 2026-02-20T04:00:02.301 INFO:scrape:suites union: ['0-subvolume/{with-no-extra-options}', '0-use-snaprealm/{subvol}', '1-ceph', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '2-workunit/iozone}}', '3-kernel}', '3-snaps/no', '4-flush/yes', '5-quiesce/no', '6-workunit/suites/iozone}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rpm_latest}', 'distro/{ubuntu/{overrides', 'frag', 'fs/mixed-clients/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'ranks/1', 'rpm_latest}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'ubuntu_latest}}', 'wsync/yes}'] 2026-02-20T04:00:02.301 INFO:scrape: 2026-02-20T04:00:02.301 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on trial099 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2026-02-20T04:00:02.301 INFO:scrape:1 jobs: ['58467'] 2026-02-20T04:00:02.301 INFO:scrape:suites: ['0-subvolume/{with-no-extra-options}', '0-use-snaprealm/{global}', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-kernel}', '3-snaps/no', '4-flush/yes', '5-quiesce/no', '6-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'pg_health', 'ranks/multi/{balancer/random', 'replication/always}', 'rpm_latest}', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'wsync/yes}'] 2026-02-20T04:00:02.301 INFO:scrape: 2026-02-20T04:00:02.301 INFO:scrape:Failure: "grep: /var/log/ceph/396a312c-0ddd-11f1-8ac1-d404e6e7d460/ceph.log: No such file or directory" in cluster log 2026-02-20T04:00:02.301 INFO:scrape:1 jobs: ['58493'] 2026-02-20T04:00:02.301 INFO:scrape:suites: ['0-subvolume/{with-namespace-isolated}', '0-use-snaprealm/{global}', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-kernel}', '3-snaps/yes', '4-flush/no', '5-quiesce/no', '6-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/default}', 'rpm_latest}', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'wsync/yes}'] 2026-02-20T04:00:02.301 INFO:scrape: 2026-02-20T04:00:02.301 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic) 2026-02-20T04:00:02.301 INFO:scrape:1 jobs: ['58349'] 2026-02-20T04:00:02.302 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/functional/{0-use-snaprealm/{subvol}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health}', 'rpm_latest}', 'tasks/xfstests-dev}'] 2026-02-20T04:00:02.302 INFO:scrape: 2026-02-20T04:00:02.302 INFO:scrape:Failure: error during thrashing 2026-02-20T04:00:02.302 INFO:scrape:4 jobs: ['58438', '58471', '58351', '58375'] 2026-02-20T04:00:02.302 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-kernel}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'pg_health', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2026-02-20T04:00:02.302 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '3-kernel}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rpm_latest}', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'pg_health', 'rpm_latest}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2026-02-20T04:00:02.302 INFO:scrape: 2026-02-20T04:00:02.302 INFO:scrape:Failure: HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=wip-rocky10-branch-of-the-day-2026-02-06-1770413686 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) 2026-02-20T04:00:02.302 INFO:scrape:3 jobs: ['58520', '58425', '58409'] 2026-02-20T04:00:02.302 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-kernel}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no_client_pidfile', 'osd}', 'overrides/{ignorelist_health', 'pg_health}'] 2026-02-20T04:00:02.302 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'distro/{ubuntu_latest}', 'fs/functional/{0-use-snaprealm/{subvol}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health}', 'rpm_latest}', 'tasks/journal-repair}', 'tasks/volumes/{0-use-snaprealm/{global}', 'test/snapshot}}'] 2026-02-20T04:00:02.302 INFO:scrape: 2026-02-20T04:00:02.302 INFO:scrape:Failure: Command failed on trial010 with status 1: 'sudo yum install -y kernel' 2026-02-20T04:00:02.302 INFO:scrape:1 jobs: ['58424'] 2026-02-20T04:00:02.302 INFO:scrape:suites: ['0-subvolume/{with-namespace-isolated-and-quota}', '0-use-snaprealm/{subvol}', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-kernel}', '3-snaps/no', '4-flush/yes', '5-quiesce/no', '6-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'pg_health', 'ranks/multi/{balancer/random', 'replication/default}', 'rpm_latest}', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'wsync/yes}'] 2026-02-20T04:00:02.302 INFO:scrape: 2026-02-20T04:00:02.302 INFO:scrape:Failure: Test failure: test_extra_data_pool_rebuild_many_files_many_workers (tasks.cephfs.test_data_scan.TestDataScan.test_extra_data_pool_rebuild_many_files_many_workers) 2026-02-20T04:00:02.302 INFO:scrape:1 jobs: ['58405'] 2026-02-20T04:00:02.302 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/functional/{0-use-snaprealm/{global}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/data-scan}'] 2026-02-20T04:00:02.302 INFO:scrape: 2026-02-20T04:00:02.302 INFO:scrape:Failure: Test failure: test_clone_after_subvol_is_removed (tasks.cephfs.volumes.test_clone_stats.TestCloneProgressReporter.test_clone_after_subvol_is_removed) 2026-02-20T04:00:02.302 INFO:scrape:1 jobs: ['58504'] 2026-02-20T04:00:02.302 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/volumes/{0-use-snaprealm/{subvol}', 'test/clone-progress}}'] 2026-02-20T04:00:02.302 INFO:scrape: 2026-02-20T04:00:02.302 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring.test_cephfs_mirror_restart_sync_on_blocklist) 2026-02-20T04:00:02.302 INFO:scrape:1 jobs: ['58455'] 2026-02-20T04:00:02.302 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'conf/{client', 'fs/mirror/{begin/{0-install', 'mds', 'mgr', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'supported-random-distros$/{rpm_latest}', 'tasks/mirror}'] 2026-02-20T04:00:02.302 INFO:scrape: 2026-02-20T04:00:02.303 INFO:scrape:Failure: Command failed on trial054 with status 6: 'curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-rocky10-branch-of-the-day-2026-02-06-1770413686/a1ac97f8290bb2544f44f608df68f43689827c55/rocky/10/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm' 2026-02-20T04:00:02.303 INFO:scrape:1 jobs: ['58416'] 2026-02-20T04:00:02.303 INFO:scrape:suites: ['0-subvolume/{with-namespace-isolated}', '0-use-snaprealm/{global}', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-kernel}', '3-snaps/no', '4-flush/yes', '5-quiesce/no', '6-workunit/suites/fsync-tester}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'pg_health', 'ranks/1', 'rpm_latest}', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'wsync/yes}'] 2026-02-20T04:00:02.303 INFO:scrape: 2026-02-20T04:00:02.303 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt_cli_setup.sh) on trial166 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt_cli_setup.sh' 2026-02-20T04:00:02.303 INFO:scrape:1 jobs: ['58439'] 2026-02-20T04:00:02.303 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'rpm_latest}', 'tasks/{1-tests/fscrypt-ffsb}}'] 2026-02-20T04:00:02.303 INFO:scrape: 2026-02-20T04:00:02.303 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ea) [0x7fc12fd4f945] 2026-02-20T04:00:02.303 INFO:scrape:ceph version 20.3.0-5184-ga1ac97f8 (a1ac97f8290bb2544f44f608df68f43689827c55) tentacle (dev - RelWithDebInfo) 1: /lib64/libc.so.6(+0x41000) [0x7fc12f669000] 2: /lib64/libc.so.6(+0x96f8c) [0x7fc12f6bef8c] 3: gsignal() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ea) [0x7fc12fd4f945] 6: ceph-mds(+0xecb6e) [0x556749c90b6e] 7: (CInode::pop_and_dirty_projected_inode(std::shared_ptr const&, boost::intrusive_ptr const&)+0x210) [0x556749f9ed20] 8: (MutationImpl::apply()+0x93) [0x556749e1b823] 9: (Server::_unlink_local_finish(boost::intrusive_ptr const&, CDentry*, CDentry*, unsigned long)+0x1db) [0x556749dd472b] 10: (MDSIOContextBase::complete(int)+0x37c) [0x55674a01ffac] 11: (MDSLogContextBase::complete(int)+0x53) [0x55674a0200f3] 12: (Finisher::finisher_thread_entry()+0x18d) [0x7fc12fe82c2d] 13: /lib64/libc.so.6(+0x95128) [0x7fc12f6bd128] 14: /lib64/libc.so.6(+0x105afc) [0x7fc12f72dafc] 2026-02-20T04:00:02.303 INFO:scrape:1 jobs: ['58358'] 2026-02-20T04:00:02.303 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'rpm_latest}', 'tasks/{1-tests/fscrypt-pjd}}'] 2026-02-20T04:00:02.303 INFO:scrape: 2026-02-20T04:00:02.303 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on trial059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2026-02-20T04:00:02.303 INFO:scrape:1 jobs: ['58371'] 2026-02-20T04:00:02.303 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rpm_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/libcephfs/{frag', 'test}}'] 2026-02-20T04:00:02.303 INFO:scrape: 2026-02-20T04:00:02.303 INFO:scrape:Failure: Test failure: test_random (tasks.cephfs.test_dump_tree.TestDumpTree.test_random) 2026-02-20T04:00:02.303 INFO:scrape:1 jobs: ['58540'] 2026-02-20T04:00:02.303 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/functional/{0-use-snaprealm/{subvol}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health}', 'rpm_latest}', 'tasks/asok_dump_tree}'] 2026-02-20T04:00:02.303 INFO:scrape: 2026-02-20T04:00:02.303 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on trial070 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 5h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2026-02-20T04:00:02.303 INFO:scrape:1 jobs: ['58487'] 2026-02-20T04:00:02.303 INFO:scrape:suites: ['0-subvolume/{with-quota}', '0-use-snaprealm/{subvol}', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-kernel}', '3-snaps/yes', '4-flush/no', '5-quiesce/no', '6-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'pg_health', 'ranks/multi/{balancer/random', 'replication/default}', 'rpm_latest}', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'wsync/yes}'] 2026-02-20T04:00:02.303 INFO:scrape: 2026-02-20T04:00:02.303 INFO:scrape:Failure: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring.test_add_ancestor_and_child_directory) 2026-02-20T04:00:02.303 INFO:scrape:1 jobs: ['58463'] 2026-02-20T04:00:02.303 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/{ignorelist_health', 'pg_health}', 'rpm_latest}', 'tasks/mirror}'] 2026-02-20T04:00:02.304 INFO:scrape: 2026-02-20T04:00:02.304 INFO:scrape:Failure: Test failure: test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction) 2026-02-20T04:00:02.304 INFO:scrape:1 jobs: ['58389'] 2026-02-20T04:00:02.304 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/volumes/{0-use-snaprealm/{subvol}', 'test/misc}}'] 2026-02-20T04:00:02.304 INFO:scrape: 2026-02-20T04:00:02.304 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs.test_acls) 2026-02-20T04:00:02.304 INFO:scrape:1 jobs: ['58527'] 2026-02-20T04:00:02.304 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/functional/{0-use-snaprealm/{global}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health}', 'rpm_latest}', 'tasks/acls}'] 2026-02-20T04:00:02.304 INFO:scrape: 2026-02-20T04:00:02.304 INFO:scrape:Failure: Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin.test_export_pin_many) 2026-02-20T04:00:02.304 INFO:scrape:1 jobs: ['58411'] 2026-02-20T04:00:02.304 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'fs/functional/{0-use-snaprealm/{global}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg_health}', 'rpm_latest}', 'tasks/exports}'] 2026-02-20T04:00:02.304 INFO:scrape: 2026-02-20T04:00:02.309 INFO:teuthology.results:Sending results to hyelloji@redhat.com: Test Run: skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial ================================================================= info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/ logs: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/ failed: 153 dead: 6 running: 0 waiting: 0 queued: 0 passed: 42 Fail ================================================================= [58344] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/yes}} ----------------------------------------------------------------- time: 00:02:56 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58344/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58344/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58345] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:20:35 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58345/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58345/ "2026-02-19T15:32:33.828339+0000 mon.a (mon.0) 308 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58346] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/with-quiesce 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58346/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58346/ Failed to reconnect to trial041 [58347] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:02:54 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58347/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58347/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58348] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58348/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58348/ Failed to reconnect to trial130 [58349] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:05:54 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58349/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58349/ Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic) [58350] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/postgres}} ----------------------------------------------------------------- time: 00:13:22 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58350/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58350/ "2026-02-19T15:36:17.410125+0000 mon.a (mon.0) 308 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58351] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:10:35 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58351/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58351/ error during thrashing [58352] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} kclient-overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/fscrypt-kbuild} ----------------------------------------------------------------- time: 04:14:30 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58352/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58352/ Command failed (workunit test fs/fscrypt_kclient_fuse.sh) on trial064 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/fscrypt-fuse && cd -- /home/ubuntu/cephtest/mnt.0/fscrypt-fuse && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 4h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/fscrypt_kclient_fuse.sh fuse' [58355] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:13:19 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58355/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58355/ "2026-02-19T15:41:07.985264+0000 mon.a (mon.0) 411 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58356] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:02:55 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58356/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58356/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58357] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:13:11 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58357/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58357/ "2026-02-19T15:41:59.577559+0000 mon.a (mon.0) 310 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58358] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-kernel} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{1-tests/fscrypt-pjd}} ----------------------------------------------------------------- time: 00:30:27 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58358/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58358/ Command failed (workunit test fs/fscrypt.sh) on trial186 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh unlocked pjd' [58359] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58359/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58359/ Failed to reconnect to trial014 [58361] fs/upgrade/nofs/{bluestore-bitmap conf/{client mds mgr mon osd} distro kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/reef 1-upgrade}} ----------------------------------------------------------------- time: 00:02:54 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58361/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58361/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58362] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:13:14 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58362/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58362/ "2026-02-19T15:46:45.838179+0000 mon.a (mon.0) 309 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58364] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58364/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58364/ Failed to reconnect to trial095 [58365] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:02:54 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58365/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58365/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58366] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:13:39 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58366/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58366/ "2026-02-19T20:53:51.795302+0000 mon.a (mon.0) 396 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58367] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:15:30 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58367/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58367/ "2026-02-19T21:01:53.751105+0000 mon.a (mon.0) 307 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58368] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:31:20 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58368/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58368/ Command failed (workunit test fs/snaps/snaptest-snap-rm-cmp.sh) on trial055 with status 4: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap- rm-cmp.sh' [58369] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58369/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58369/ Failed to reconnect to trial073 [58370] fs/bugs/multifs_mdsauthcaps/{clusters/1node-cluster-1node-client conf/{client mds mgr mon osd} distro/{rpm_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/{0-install 1-mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} 2-test_admin_squid_client}} ----------------------------------------------------------------- time: 00:02:58 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58370/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58370/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&sha1=0ec eb0defba60152a8182f7bd87d164b639885b8 [58371] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{rpm_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:51:07 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58371/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58371/ Command failed (workunit test libcephfs/test.sh) on trial059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [58373] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:13:12 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58373/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58373/ "2026-02-19T20:57:55.832479+0000 mon.a (mon.0) 309 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58374] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:02:55 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58374/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58374/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58375] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:05:51 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58375/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58375/ error during thrashing [58377] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/backtrace} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58377/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58377/ Failed to reconnect to trial104 [58378] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:13:17 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58378/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58378/ "2026-02-19T21:02:57.468800+0000 mon.a (mon.0) 413 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58380] fs/upgrade/upgraded_client/{bluestore-bitmap clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58380/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58380/ Failed to reconnect to trial172 [58381] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:13:11 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58381/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58381/ "2026-02-19T21:04:55.225527+0000 mon.a (mon.0) 311 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58382] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58382/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58382/ Failed to reconnect to trial134 [58383] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58383/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58383/ Failed to reconnect to trial046 [58384] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:02:54 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58384/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58384/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58385] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-kernel} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58385/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58385/ Failed to reconnect to trial130 [58386] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58386/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58386/ Failed to reconnect to trial048 [58389] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{0-use-snaprealm/{subvol} overrides test/misc}} ----------------------------------------------------------------- time: 00:09:12 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58389/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58389/ Test failure: test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction) [58390] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/direct_io}} ----------------------------------------------------------------- time: 00:11:45 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58390/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58390/ "2026-02-19T21:08:42.000502+0000 mon.a (mon.0) 309 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58392] fs/upgrade/upgraded_client/{bluestore-bitmap clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/reef 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}} ----------------------------------------------------------------- time: 00:02:53 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58392/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58392/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58393] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:14:06 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58393/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58393/ "2026-02-19T21:16:57.968186+0000 mon.a (mon.0) 410 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58394] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client-limits} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58394/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58394/ Failed to reconnect to trial170 [58395] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:13:57 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58395/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58395/ "2026-02-19T21:11:42.033550+0000 mon.a (mon.0) 305 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58396] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/yes}} ----------------------------------------------------------------- time: 00:02:59 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58396/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58396/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58397] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58397/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58397/ Failed to reconnect to trial092 [58399] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:12:53 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58399/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58399/ Command failed (workunit test suites/blogbench.sh) on trial135 with status 4: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh' [58400] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:02:54 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58400/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58400/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&sha1=5dd 24139a1eada541a3bc16b6941c5dde975e26d [58401] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58401/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58401/ Failed to reconnect to trial060 [58402] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:45:25 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58402/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58402/ "2026-02-19T21:17:09.511442+0000 mon.a (mon.0) 399 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58403] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/damage} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58403/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58403/ Failed to reconnect to trial193 [58404] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:13:22 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58404/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58404/ "2026-02-19T21:18:03.551092+0000 mon.a (mon.0) 310 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58405] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/data-scan} ----------------------------------------------------------------- time: 00:08:07 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58405/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58405/ Test failure: test_extra_data_pool_rebuild_many_files_many_workers (tasks.c ephfs.test_data_scan.TestDataScan.test_extra_data_pool_rebuild_many_files_m any_workers) [58407] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58407/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58407/ Failed to reconnect to trial027 [58408] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:03:02 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58408/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58408/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58409] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{0-use-snaprealm/{global} overrides test/snapshot}} ----------------------------------------------------------------- time: 00:03:55 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58409/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58409/ HTTPSConnectionPool(host='2.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/wip-rocky10-branch-of-the-day-2026-02-06- 1770413686/a1ac97f8290bb2544f44f608df68f43689827c55/rocky/10/flavors/defaul t/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) [58410] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:15:21 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58410/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58410/ "2026-02-19T21:22:58.364270+0000 mon.a (mon.0) 311 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58411] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/exports} ----------------------------------------------------------------- time: 00:52:40 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58411/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58411/ Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin.test_export_pin_many) [58412] fs/upgrade/nofs/{bluestore-bitmap conf/{client mds mgr mon osd} distro kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}} ----------------------------------------------------------------- time: 00:02:56 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58412/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58412/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58413] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:03:56 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58413/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58413/ Command failed on trial171 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph- mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-osd ceph-osd-classic ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs- top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib' [58414] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58414/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58414/ Failed to reconnect to trial199 [58415] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58415/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58415/ Failed to reconnect to trial082 [58416] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:05:30 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58416/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58416/ Command failed on trial054 with status 6: 'curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-rocky10-branch-of-the-day-2026- 02-06- 1770413686/a1ac97f8290bb2544f44f608df68f43689827c55/rocky/10/x86_64/flavors /default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm' [58418] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58418/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58418/ Failed to reconnect to trial009 [58419] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/exports 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58419/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58419/ Failed to reconnect to trial072 [58420] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} kclient-overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 03:15:02 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58420/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58420/ Command failed (workunit test suites/blogbench.sh) on trial073 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh' [58421] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58421/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58421/ Failed to reconnect to trial126 [58422] fs/snaps/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58422/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58422/ Failed to reconnect to trial144 [58423] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:02:57 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58423/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58423/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58424] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58424/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58424/ Command failed on trial010 with status 1: 'sudo yum install -y kernel' [58425] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/journal-repair} ----------------------------------------------------------------- time: 00:03:59 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58425/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58425/ HTTPSConnectionPool(host='2.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/wip-rocky10-branch-of-the-day-2026-02-06- 1770413686/a1ac97f8290bb2544f44f608df68f43689827c55/rocky/10/flavors/defaul t/repo (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) [58426] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:16:27 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58426/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58426/ "2026-02-19T21:31:13.101397+0000 mon.a (mon.0) 407 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58428] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:03:37 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58428/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58428/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58429] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:15:57 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58429/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58429/ "2026-02-19T21:33:20.242524+0000 mon.a (mon.0) 297 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58432] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/direct_io}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58432/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58432/ Failed to reconnect to trial010 [58433] fs/upgrade/upgraded_client/{bluestore-bitmap clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58433/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58433/ Failed to reconnect to trial114 [58434] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:15:13 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58434/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58434/ "2026-02-19T21:33:51.901650+0000 mon.a (mon.0) 306 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58436] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:04:37 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58436/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58436/ Command 'git ls-remote https://github.com/ceph/ceph-ci v18.2.1^{}' returned non-zero exit status 128. [58437] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:14:42 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58437/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58437/ "2026-02-19T21:34:51.800999+0000 mon.a (mon.0) 416 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58438] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:08:49 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58438/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58438/ error during thrashing [58439] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-kernel} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{1-tests/fscrypt-ffsb}} ----------------------------------------------------------------- time: 00:07:46 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58439/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58439/ Command failed (workunit test fs/fscrypt_cli_setup.sh) on trial166 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt_cli_setup.sh' [58440] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/postgres}} ----------------------------------------------------------------- time: 00:30:12 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58440/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58440/ "2026-02-19T21:34:55.982066+0000 mon.a (mon.0) 303 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58441] fs/upgrade/upgraded_client/{bluestore-bitmap clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/reef 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} ----------------------------------------------------------------- time: 00:03:35 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58441/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58441/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58442] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58442/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58442/ Failed to reconnect to trial061 [58445] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/yes}} ----------------------------------------------------------------- time: 00:03:19 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58445/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58445/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58446] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:16:03 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58446/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58446/ "2026-02-19T21:37:39.883532+0000 mon.a (mon.0) 301 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58448] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:13:37 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58448/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58448/ "2026-02-19T21:35:06.906701+0000 mon.a (mon.0) 405 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58449] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58449/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58449/ Command failed on trial183 with status 4: 'rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.17.0_gd74d6c0e9895-2.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://chacra.ceph.com/r/kernel/testing/d74d 6c0e98958aa0bdb6f0a93258a856bda58b97/rocky/10/flavors/default/x86_64/ --input-file=-' [58450] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:14:27 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58450/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58450/ "2026-02-19T21:38:34.001072+0000 mon.a (mon.0) 302 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58455] fs/mirror/{begin/{0-install 1-ceph 2-logrotate 3-kernel} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} conf/{client mds mgr mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{rpm_latest} tasks/mirror} ----------------------------------------------------------------- time: 01:17:14 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58455/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58455/ Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.te st_mirroring.TestMirroring.test_cephfs_mirror_restart_sync_on_blocklist) [58457] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/fscrypt-kbuild} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58457/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58457/ Failed to reconnect to trial020 [58461] fs/snaps/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:15:11 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58461/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58461/ Command failed (workunit test fs/snaps/snaptest-snap-rm-cmp.sh) on trial046 with status 4: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap- rm-cmp.sh' [58463] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate 3-kernel} debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/{ignorelist_health pg_health} tasks/mirror} rpm_latest} ----------------------------------------------------------------- time: 00:11:17 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58463/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58463/ Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirr oring.TestMirroring.test_add_ancestor_and_child_directory) [58464] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:03:15 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58464/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58464/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58465] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58465/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58465/ Failed to reconnect to trial075 [58467] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:13:01 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58467/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58467/ Command failed (workunit test suites/fsx.sh) on trial099 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [58468] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:03:48 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58468/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58468/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58469] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:27:55 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58469/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58469/ "2026-02-19T21:44:39.400599+0000 mon.a (mon.0) 411 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58471] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:19:18 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58471/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58471/ error during thrashing [58472] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:16:29 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58472/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58472/ "2026-02-19T21:50:06.148496+0000 mon.a (mon.0) 305 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58473] fs/upgrade/nofs/{bluestore-bitmap conf/{client mds mgr mon osd} distro kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/reef 1-upgrade}} ----------------------------------------------------------------- time: 00:03:14 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58473/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58473/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58474] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58474/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58474/ Failed to reconnect to trial195 [58475] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58475/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58475/ Failed to reconnect to trial057 [58476] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/no 6-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:14:22 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58476/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58476/ Command failed (workunit test suites/iozone.sh) on trial006 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh' [58477] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:27:56 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58477/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58477/ "2026-02-19T21:50:41.199545+0000 mon.a (mon.0) 405 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58479] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:03:48 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58479/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58479/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58480] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:07:33 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58480/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58480/ Command failed (workunit test suites/iozone.sh) on trial164 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/iozone.sh' [58481] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/direct_io}} ----------------------------------------------------------------- time: 00:15:29 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58481/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58481/ "2026-02-19T21:52:18.021500+0000 mon.a (mon.0) 395 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58482] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:03:11 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58482/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58482/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58485] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58485/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58485/ Failed to reconnect to trial026 [58487] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:29:55 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58487/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58487/ Command failed (workunit test kernel_untar_build.sh) on trial070 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 5h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [58488] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} ----------------------------------------------------------------- time: 00:14:55 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58488/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58488/ "2026-02-19T21:53:22.544406+0000 mon.a (mon.0) 313 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58490] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 03:13:29 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58490/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58490/ Command failed (workunit test suites/iozone.sh) on trial072 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1ac97f8290bb2544f44f608df68f43689827c55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh' [58493] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:05:17 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58493/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58493/ "grep: /var/log/ceph/396a312c-0ddd-11f1-8ac1-d404e6e7d460/ceph.log: No such file or directory" in cluster log [58494] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58494/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58494/ Command failed on trial007 with status 4: 'rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.17.0_gd74d6c0e9895-2.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://chacra.ceph.com/r/kernel/testing/d74d 6c0e98958aa0bdb6f0a93258a856bda58b97/rocky/10/flavors/default/x86_64/ --input-file=-' [58495] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:03:04 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58495/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58495/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58496] fs/bugs/multifs_mdsauthcaps/{clusters/1node-cluster-1node-client conf/{client mds mgr mon osd} distro/{rpm_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/{0-install 1-mount/fuse 2-test_admin_squid_client}} ----------------------------------------------------------------- time: 00:02:49 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58496/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58496/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&sha1=0ec eb0defba60152a8182f7bd87d164b639885b8 [58497] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-kernel} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{1-tests/fscrypt-pjd}} ----------------------------------------------------------------- time: 00:03:53 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58497/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58497/ Command failed on trial190 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph- mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-osd ceph-osd-classic ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs- top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib' [58499] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:14:36 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58499/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58499/ "2026-02-19T21:56:21.498149+0000 mon.a (mon.0) 411 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58500] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:08:46 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58500/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58500/ Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.T estJournalMigration.test_journal_migration) [58501] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58501/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58501/ Failed to reconnect to trial033 [58502] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:15:48 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58502/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58502/ "2026-02-19T21:57:12.294238+0000 mon.a (mon.0) 302 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58503] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/yes}} ----------------------------------------------------------------- time: 00:03:34 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58503/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58503/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58504] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{0-use-snaprealm/{subvol} overrides test/clone-progress}} ----------------------------------------------------------------- time: 00:09:15 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58504/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58504/ Test failure: test_clone_after_subvol_is_removed (tasks.cephfs.volumes.test _clone_stats.TestCloneProgressReporter.test_clone_after_subvol_is_removed) [58506] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:15:50 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58506/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58506/ "2026-02-19T21:59:31.888896+0000 mon.a (mon.0) 400 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58507] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58507/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58507/ Command failed on trial142 with status 4: 'rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.17.0_gd74d6c0e9895-2.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://chacra.ceph.com/r/kernel/testing/d74d 6c0e98958aa0bdb6f0a93258a856bda58b97/rocky/10/flavors/default/x86_64/ --input-file=-' [58508] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58508/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58508/ Failed to reconnect to trial044 [58510] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:13:37 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58510/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58510/ "2026-02-19T22:00:33.524497+0000 mon.a (mon.0) 398 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58511] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:03:20 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58511/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58511/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58512] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:13:56 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58512/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58512/ "2026-02-19T22:00:27.498973+0000 mon.a (mon.0) 306 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58514] fs/upgrade/nofs/{bluestore-bitmap conf/{client mds mgr mon osd} distro kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}} ----------------------------------------------------------------- time: 00:03:02 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58514/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58514/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58516] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:13:29 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58516/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58516/ "2026-02-19T22:01:34.435081+0000 mon.a (mon.0) 405 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58517] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58517/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58517/ Failed to reconnect to trial061 [58518] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58518/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58518/ Failed to reconnect to trial038 [58519] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:02:53 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58519/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58519/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef [58520] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58520/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58520/ HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=rock y%2F10%2Fx86_64&ref=wip-rocky10-branch-of-the-day-2026-02-06-1770413686 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) [58521] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:13:26 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58521/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58521/ "2026-02-19T22:02:29.998244+0000 mon.a (mon.0) 398 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58523] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58523/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58523/ Failed to reconnect to trial196 [58524] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/direct_io}} ----------------------------------------------------------------- time: 00:17:54 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58524/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58524/ "2026-02-19T22:03:26.842418+0000 mon.a (mon.0) 302 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58526] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:13:10 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58526/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58526/ "2026-02-19T22:03:56.854720+0000 mon.a (mon.0) 402 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58527] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/acls} ----------------------------------------------------------------- time: 00:05:28 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58527/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58527/ Test failure: test_acls (tasks.cephfs.test_acls.TestACLs.test_acls) [58528] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:02:52 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58528/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58528/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58529] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58529/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58529/ Failed to reconnect to trial039 [58530] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58530/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58530/ Failed to reconnect to trial040 [58532] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/postgres}} ----------------------------------------------------------------- time: 00:13:29 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58532/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58532/ "2026-02-19T22:07:36.387194+0000 mon.a (mon.0) 397 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58533] fs/upgrade/upgraded_client/{bluestore-bitmap clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}} ----------------------------------------------------------------- time: 00:02:50 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58533/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58533/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58535] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:13:21 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58535/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58535/ "2026-02-19T22:08:05.706515+0000 mon.a (mon.0) 313 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58536] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap conf/{client mds mgr mon osd} distro fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:02:52 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58536/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58536/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=squi d [58538] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:20:32 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58538/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58538/ valgrind error: MismatchedFree operator delete[](void*, unsigned long, std::align_val_t) UnknownInlinedFun UnknownInlinedFun [58539] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:13:31 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58539/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58539/ "2026-02-19T22:09:19.251349+0000 mon.a (mon.0) 399 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58540] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:10:40 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58540/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58540/ Test failure: test_random (tasks.cephfs.test_dump_tree.TestDumpTree.test_random) [58541] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58541/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58541/ Failed to reconnect to trial075 [58542] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/auto-repair} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58542/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58542/ Failed to reconnect to trial074 [58543] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:13:32 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58543/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58543/ "2026-02-19T22:11:04.109561+0000 mon.a (mon.0) 412 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [58544] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} distro kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/yes}} ----------------------------------------------------------------- time: 00:02:54 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58544/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58544/ Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef Dead ================================================================= [58360] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58360/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58360/ hit max job timeout [58398] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client-recovery} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58398/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58398/ hit max job timeout [58466] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/quiesce} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58466/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58466/ killed [58470] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/quota} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58470/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58470/ hit max job timeout [58478] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/scrub} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58478/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58478/ hit max job timeout [58522] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-kernel} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58522/ log: https://qa-proxy.ceph.com/teuthology/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58522/ killed Pass ================================================================= [58353] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:28:07 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58353/ [58354] fs/snaps/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:28:35 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58354/ [58363] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{0-use-snaprealm/{subvol} overrides test/metrics}} time: 00:07:47 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58363/ [58372] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/auto-repair} time: 00:07:37 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58372/ [58376] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/dbench validater/lockdep} time: 00:42:55 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58376/ [58379] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:06:45 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58379/ [58387] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:25:35 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58387/ [58388] fs/snaps/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:13:49 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58388/ [58391] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:13:39 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58391/ [58406] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_oft_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/with-quiesce 2-workunit/fs/trivial_sync}} time: 00:05:56 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58406/ [58417] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/fragment} time: 00:28:58 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58417/ [58427] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:12:44 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58427/ [58430] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:08:27 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58430/ [58431] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:07:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58431/ [58435] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/mds_creation_retry} time: 00:09:02 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58435/ [58443] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_oft_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:59:42 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58443/ [58444] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/multimds_misc} time: 00:12:22 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58444/ [58447] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/openfiletable} time: 00:10:31 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58447/ [58451] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:07:10 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58451/ [58452] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/small-cluster conf/{client mds mgr mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} rpm_latest tasks/trim-i24137} time: 00:07:08 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58452/ [58453] fs/full/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1-node-4-mds-1-osd conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/mgr-osd-full} time: 00:14:30 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58453/ [58454] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{rpm_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client} time: 00:09:28 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58454/ [58456] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate 3-kernel} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} conf/{client mds mgr mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distro$/{rpm_latest} workloads/cephfs-mirror-ha-workunit} time: 00:14:38 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58456/ [58458] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:24:12 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58458/ [58459] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:07:10 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58459/ [58460] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/rpm_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} time: 00:29:56 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58460/ [58462] fs/top/{begin/{0-install 1-ceph 2-logrotate 3-kernel} cluster/{1-node} conf/{client mds mgr mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{rpm_latest} tasks/fstop} time: 00:08:02 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58462/ [58483] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:07:36 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58483/ [58484] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:07:24 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58484/ [58486] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/exports 2-workunit/fs/trivial_sync}} time: 00:06:38 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58486/ [58489] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/snapshots} time: 00:34:27 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58489/ [58491] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:33:39 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58491/ [58492] fs/snaps/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:18:19 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58492/ [58498] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{rpm_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client_full} time: 00:09:26 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58498/ [58505] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/test_snap_schedule/{overrides/{v1} snap-schedule}} time: 02:28:10 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58505/ [58509] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/no prefetch_oft_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:07:58 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58509/ [58513] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:07:47 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58513/ [58515] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rpm_latest} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/workunit/dir-max-entries} time: 00:05:59 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58515/ [58525] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:07:01 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58525/ [58531] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:06:57 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58531/ [58534] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/alternate-pool} time: 00:06:05 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58534/ [58537] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:21:00 info: https://pulpito.ceph.com/skanta-2026-02-19_12:04:07-fs-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/58537/ 2026-02-20T04:00:02.317 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 95, in email_results smtp = smtplib.SMTP('localhost') ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 255, in __init__ (code, msg) = self.connect(host, port) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 341, in connect self.sock = self._get_socket(host, port, self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket return socket.create_connection((host, port), timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection sock.connect(sa) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect raise _SocketError(err, strerror(err)) ConnectionRefusedError: [Errno 111] Connection refused