2026-02-20T16:07:05.468 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-20T16:07:05.468 INFO:teuthology.results:subset: '111/120000' 2026-02-20T16:07:05.469 INFO:teuthology.results:seed: '6167' 2026-02-20T17:07:04.855 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-20T17:07:04.862 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-20T21:29:52.305 INFO:teuthology.results:Tests finished! gathering results... 2026-02-20T21:29:54.204 INFO:scrape:Found 69 jobs 2026-02-20T21:29:56.676 INFO:scrape:Found 6 distinct failure reasons 2026-02-20T21:29:56.676 INFO:scrape:Failure: "2026-02-20T18:00:00.000171+0000 mon.a (mon.0) 12513 : cluster [WRN] pg 4.1a is stuck peering for 10m, current state peering, last acting [2,4,0]" in cluster log 2026-02-20T21:29:56.676 INFO:scrape:13 jobs: ['60848', '60846', '60894', '60817', '60805', '60904', '60874', '60886', '60918', '60834', '60888', '60856', '60798'] 2026-02-20T21:29:56.676 INFO:scrape:suites intersection: [] 2026-02-20T21:29:56.676 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks}', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/connectivity', 'overrides/ignorelist_health}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/squid-x/stress-split/{0-distro/centos_9.stream', 'upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/telemetry-upgrade/reef-x/{0-random-distro$/{centos_9.stream_runc}'] 2026-02-20T21:29:56.676 INFO:scrape: 2026-02-20T21:29:56.676 INFO:scrape:Failure: Command failed (workunit test test_telemetry_squid.sh) on trial009 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' 2026-02-20T21:29:56.676 INFO:scrape:5 jobs: ['60809', '60880', '60840', '60806', '60910'] 2026-02-20T21:29:56.676 INFO:scrape:suites intersection: ['0-start'] 2026-02-20T21:29:56.676 INFO:scrape:suites union: ['0-start', '1-tasks', '1-tasks}', 'mon_election/classic', 'mon_election/connectivity', 'overrides/ignorelist_health', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/squid-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/telemetry-upgrade/squid-x/{0-random-distro$/{centos_9.stream}', 'workload/{ec-rados-default'] 2026-02-20T21:29:56.676 INFO:scrape: 2026-02-20T21:29:56.676 INFO:scrape:Failure: 2026-02-20T21:29:56.676 INFO:scrape:3 jobs: ['60821', '60890', '60850'] 2026-02-20T21:29:56.676 INFO:scrape:suites intersection: ['centos_9.stream', 'conf/{client', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}'] 2026-02-20T21:29:56.676 INFO:scrape:suites union: ['1-client', '1-mount/mount/kclient/{mount-syntax/{v1}', '2-clients/kclient', '2-upgrade', '3-client-upgrade', '3-workload/stress_tests/fsstress}}', '4-compat_client', '5-client-sanity}}', 'centos_9.stream', 'clusters/1-mds-2-client-micro', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-from/reef', 'tasks/{0-install/reef', 'upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap', 'upgrade/cephfs/upgraded_client/{bluestore-bitmap'] 2026-02-20T21:29:56.676 INFO:scrape: 2026-02-20T21:29:56.676 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2026-02-20T21:29:56.676 INFO:scrape:6 jobs: ['60866', '60906', '60813', '60836', '60882', '60799'] 2026-02-20T21:29:56.676 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2026-02-20T21:29:56.677 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2026-02-20T21:29:56.677 INFO:scrape: 2026-02-20T21:29:56.677 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/fsstress.sh 2026-02-20T21:29:56.677 INFO:scrape:1 jobs: ['60892'] 2026-02-20T21:29:56.677 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/squid', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2026-02-20T21:29:56.677 INFO:scrape: 2026-02-20T21:29:56.677 INFO:scrape:Timeout 5h running clone.client.0/qa/workunits/kernel_untar_build.sh 2026-02-20T21:29:56.677 INFO:scrape:1 jobs: ['60819'] 2026-02-20T21:29:56.677 INFO:scrape:suites: ['1-mount/mount/kclient/{mount-syntax/{v1}', '2-clients/kclient', '3-workload/stress_tests/kernel_untar_build}}', 'centos_9.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-install/reef', 'upgrade/cephfs/upgraded_client/{bluestore-bitmap'] 2026-02-20T21:29:56.677 INFO:scrape: 2026-02-20T21:29:56.679 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial ================================================================= info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/ logs: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/ failed: 29 dead: 0 running: 0 waiting: 0 queued: 0 passed: 40 Fail ================================================================= [60798] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:15:49 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60798/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60798/ "2026-02-20T17:10:00.000140+0000 mon.a (mon.0) 4441 : cluster [WRN] pg 4.7 is stuck peering for 61s, current state peering, last acting [5,0,7]" in cluster log [60799] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:13 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60799/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60799/ reached maximum tries (50) after waiting for 300 seconds [60805] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 01:50:47 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60805/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60805/ "2026-02-20T16:40:00.000180+0000 mon.a (mon.0) 2954 : cluster [WRN] pg 2.1 is stuck peering for 10m, current state peering, last acting [0,2,1]" in cluster log [60806] upgrade/telemetry-upgrade/squid-x/{0-random-distro$/{centos_9.stream} 0-start 1-tasks} ----------------------------------------------------------------- time: 00:08:10 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60806/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60806/ Command failed (workunit test test_telemetry_squid.sh) on trial013 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' [60809] upgrade/squid-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:14:42 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60809/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60809/ Command failed (workunit test test_telemetry_squid.sh) on trial009 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' [60813] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:29 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60813/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60813/ reached maximum tries (50) after waiting for 300 seconds [60817] upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:07:29 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60817/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60817/ "2026-02-20T16:40:00.000153+0000 mon.a (mon.0) 2523 : cluster [WRN] pg 4.2 is stuck peering for 4m, current state peering, last acting [4,2,5]" in cluster log [60819] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/reef 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/kernel_untar_build}} ----------------------------------------------------------------- time: 05:12:48 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60819/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60819/ Command failed (workunit test kernel_untar_build.sh) on trial076 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 5h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [60821] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 02:21:34 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60821/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60821/ [60834] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:10:22 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60834/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60834/ "2026-02-20T17:20:00.000106+0000 mon.a (mon.0) 7654 : cluster [WRN] daemon osd.3 on trial105 is in unknown state" in cluster log [60836] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:37 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60836/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60836/ reached maximum tries (50) after waiting for 300 seconds [60840] upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:08:25 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60840/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60840/ Command failed (workunit test test_telemetry_squid.sh) on trial024 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' [60846] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:05:40 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60846/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60846/ "2026-02-20T17:07:31.808528+0000 mon.a (mon.0) 3816 : cluster [WRN] Health check failed: 3464/175293 objects unfound (1.976%) (OBJECT_UNFOUND)" in cluster log [60848] upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:08:21 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60848/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60848/ "2026-02-20T18:00:00.000171+0000 mon.a (mon.0) 12513 : cluster [WRN] pg 4.1a is stuck peering for 10m, current state peering, last acting [2,4,0]" in cluster log [60850] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/reef 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/fsstress}} ----------------------------------------------------------------- time: 02:07:10 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60850/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60850/ [60856] upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:20:49 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60856/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60856/ "2026-02-20T16:40:29.830965+0000 mon.a (mon.0) 17 : cluster [WRN] pg 4.1b is stuck peering for 66s, current state peering, last acting [4,3,2]" in cluster log [60866] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:19:45 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60866/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60866/ reached maximum tries (50) after waiting for 300 seconds [60874] upgrade/telemetry-upgrade/reef-x/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks} ----------------------------------------------------------------- time: 00:13:37 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60874/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60874/ "2026-02-20T16:48:08.194456+0000 mon.a (mon.0) 165 : cluster [WRN] Health detail: HEALTH_WARN Telemetry requires re-opt-in" in cluster log [60880] upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:08:18 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60880/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60880/ Command failed (workunit test test_telemetry_squid.sh) on trial077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' [60882] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:19:13 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60882/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60882/ reached maximum tries (50) after waiting for 300 seconds [60886] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:35:53 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60886/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60886/ "2026-02-20T17:00:00.000138+0000 mon.a (mon.0) 796 : cluster [ERR] fs foo is offline because no MDS is active for it." in cluster log [60888] upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:34:34 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60888/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60888/ "2026-02-20T19:10:00.000203+0000 mon.a (mon.0) 17000 : cluster [WRN] pg 1.7 is stuck peering for 92s, current state peering, last acting [6,3,7]" in cluster log [60890] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 02:23:41 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60890/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60890/ [60892] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 04:05:31 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60892/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60892/ Command failed (workunit test suites/fsstress.sh) on trial039 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [60894] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:16:14 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60894/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60894/ "2026-02-20T17:00:00.000112+0000 mon.a (mon.0) 770 : cluster [ERR] fs foo is offline because no MDS is active for it." in cluster log [60904] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:11:31 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60904/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60904/ "2026-02-20T17:23:28.812466+0000 mon.a (mon.0) 1811 : cluster [WRN] Health check failed: 1/119102 objects unfound (0.001%) (OBJECT_UNFOUND)" in cluster log [60906] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:20:13 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60906/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60906/ reached maximum tries (50) after waiting for 300 seconds [60910] upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:08:02 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60910/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60910/ Command failed (workunit test test_telemetry_squid.sh) on trial060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' [60918] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:18:08 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60918/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60918/ "2026-02-20T19:00:00.000095+0000 mon.a (mon.0) 15390 : cluster [WRN] pg 62.29 is stuck peering for 3m, current state remapped+peering, last acting [5,6]" in cluster log Pass ================================================================= [60796] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:11:16 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60796/ [60797] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:26:52 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60797/ [60800] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:20:17 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60800/ [60801] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:20:29 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60801/ [60802] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/yes}} time: 01:24:21 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60802/ [60803] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:14:12 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60803/ [60804] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 01:50:59 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60804/ [60807] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/no}} time: 01:29:48 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60807/ [60808] upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:49:09 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60808/ [60811] upgrade/cephfs/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/reef 1-upgrade}} time: 00:05:56 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60811/ [60815] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:01:57 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60815/ [60823] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:39:27 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60823/ [60825] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:44:28 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60825/ [60828] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:15:20 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60828/ [60830] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/yes}} time: 00:10:44 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60830/ [60832] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:02:10 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60832/ [60838] upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:49:39 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60838/ [60842] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/no}} time: 00:12:14 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60842/ [60844] upgrade/cephfs/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}} time: 00:06:35 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60844/ [60852] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:15:00 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60852/ [60854] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:17:25 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60854/ [60858] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:11:44 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60858/ [60860] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}} time: 00:08:11 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60860/ [60862] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:09:05 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60862/ [60864] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:07:33 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60864/ [60868] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/yes}} time: 01:27:55 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60868/ [60870] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:09:22 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60870/ [60872] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:06:27 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60872/ [60876] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/no}} time: 01:28:51 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60876/ [60878] upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:49:49 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60878/ [60884] upgrade/cephfs/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/reef 1-upgrade}} time: 00:05:53 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60884/ [60896] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:12:33 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60896/ [60898] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} time: 00:44:37 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60898/ [60900] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/yes}} time: 00:10:48 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60900/ [60902] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:02:11 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60902/ [60908] upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:44:30 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60908/ [60912] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/no}} time: 00:15:47 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60912/ [60914] upgrade/cephfs/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}} time: 00:06:02 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60914/ [60916] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity overrides/ignorelist_health} time: 02:16:37 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60916/ [60920] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:14:55 info: https://pulpito.ceph.com/yuriw-2026-02-20_16:06:46-upgrade-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/60920/ 2026-02-20T21:29:56.686 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 95, in email_results smtp = smtplib.SMTP('localhost') ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 255, in __init__ (code, msg) = self.connect(host, port) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 341, in connect self.sock = self._get_socket(host, port, self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket return socket.create_connection((host, port), timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection sock.connect(sa) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect raise _SocketError(err, strerror(err)) ConnectionRefusedError: [Errno 111] Connection refused