2026-02-17T21:19:51.265 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71 2026-02-17T21:19:51.265 INFO:teuthology.results:subset: '111/120000' 2026-02-17T21:19:51.265 INFO:teuthology.results:seed: '8679' 2026-02-17T22:30:41.036 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71 2026-02-17T22:30:41.042 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-18T04:34:29.177 INFO:teuthology.results:Tests finished! gathering results... 2026-02-18T04:34:31.063 INFO:scrape:Found 68 jobs 2026-02-18T04:34:33.494 INFO:scrape:Found 6 distinct failure reasons 2026-02-18T04:34:33.494 INFO:scrape:Failure: "2026-02-17T22:50:00.000138+0000 mon.a (mon.0) 5949 : cluster [WRN] pg 1.5 is stuck peering for 2m, current state remapped+peering, last acting [2,6]" in cluster log 2026-02-18T04:34:33.494 INFO:scrape:14 jobs: ['53954', '53953', '53974', '53980', '53933', '53961', '53971', '53987', '53946', '54002', '53978', '53979', '53975', '53937'] 2026-02-18T04:34:33.494 INFO:scrape:suites intersection: [] 2026-02-18T04:34:33.494 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks}', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/connectivity', 'overrides/ignorelist_health}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/squid-x/stress-split/{0-distro/centos_9.stream', 'upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/telemetry-upgrade/reef-x/{0-random-distro$/{centos_9.stream_runc}'] 2026-02-18T04:34:33.494 INFO:scrape: 2026-02-18T04:34:33.494 INFO:scrape:Failure: 2026-02-18T04:34:33.494 INFO:scrape:4 jobs: ['53955', '53977', '53981', '53989'] 2026-02-18T04:34:33.495 INFO:scrape:suites intersection: ['1-client', '2-upgrade', 'centos_9.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-from/reef'] 2026-02-18T04:34:33.495 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/yes}}', '4-compat_client', '5-client-sanity}}', 'centos_9.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-from/reef', 'upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap', 'upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap'] 2026-02-18T04:34:33.495 INFO:scrape: 2026-02-18T04:34:33.495 INFO:scrape:Failure: Command failed (workunit test test_telemetry_squid.sh) on trial125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adc1923395f2bcd6ae89b5d331d278364d1848ed TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' 2026-02-18T04:34:33.495 INFO:scrape:5 jobs: ['53964', '54008', '53947', '53983', '53950'] 2026-02-18T04:34:33.495 INFO:scrape:suites intersection: ['0-start'] 2026-02-18T04:34:33.495 INFO:scrape:suites union: ['0-start', '1-tasks', '1-tasks}', 'mon_election/classic', 'mon_election/connectivity', 'overrides/ignorelist_health', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/squid-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/telemetry-upgrade/squid-x/{0-random-distro$/{centos_9.stream}', 'workload/{ec-rados-default'] 2026-02-18T04:34:33.495 INFO:scrape: 2026-02-18T04:34:33.495 INFO:scrape:Failure: Command failed on trial080 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' 2026-02-18T04:34:33.495 INFO:scrape:1 jobs: ['53988'] 2026-02-18T04:34:33.495 INFO:scrape:suites: ['1-mount/mount/kclient/{mount-syntax/{v1}', '2-clients/kclient', '3-workload/stress_tests/blogbench}}', 'centos_9.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-install/reef', 'upgrade/cephfs/upgraded_client/{bluestore-bitmap'] 2026-02-18T04:34:33.495 INFO:scrape: 2026-02-18T04:34:33.495 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2026-02-18T04:34:33.495 INFO:scrape:6 jobs: ['54004', '53952', '53962', '53984', '53976', '53935'] 2026-02-18T04:34:33.495 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2026-02-18T04:34:33.495 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2026-02-18T04:34:33.495 INFO:scrape: 2026-02-18T04:34:33.495 INFO:scrape:Dead: b'2026-02-18T04:34:04.796 DEBUG:teuthology.exit:Finished running handlers' 2026-02-18T04:34:33.495 INFO:scrape:2 jobs: ['54016', '53970'] 2026-02-18T04:34:33.495 INFO:scrape:suites intersection: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'overrides/ignorelist_health}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2026-02-18T04:34:33.495 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/classic', 'mon_election/connectivity', 'overrides/ignorelist_health}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/squid-x/stress-split/{0-distro/centos_9.stream'] 2026-02-18T04:34:33.495 INFO:scrape: 2026-02-18T04:34:33.497 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial ================================================================= info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/ logs: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/ failed: 30 dead: 2 running: 0 waiting: 0 queued: 0 passed: 36 Fail ================================================================= [53933] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:15:00 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53933/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53933/ "2026-02-17T23:40:00.000097+0000 mon.a (mon.0) 10842 : cluster [WRN] pg 1.a is stuck peering for 4m, current state remapped+peering, last acting [5,2]" in cluster log [53935] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:05 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53935/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53935/ reached maximum tries (50) after waiting for 300 seconds [53937] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:20:56 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53937/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53937/ "2026-02-17T22:12:02.920501+0000 mon.a (mon.0) 1203 : cluster [WRN] Health check failed: 1283/166334 objects unfound (0.771%) (OBJECT_UNFOUND)" in cluster log [53946] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 01:54:54 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53946/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53946/ "2026-02-17T22:40:00.000201+0000 mon.a (mon.0) 7155 : cluster [WRN] pg 3.b is stuck peering for 2m, current state peering, last acting [5,0,1]" in cluster log [53947] upgrade/telemetry-upgrade/squid-x/{0-random-distro$/{centos_9.stream} 0-start 1-tasks} ----------------------------------------------------------------- time: 00:06:48 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53947/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53947/ Command failed (workunit test test_telemetry_squid.sh) on trial190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adc1923395f2bcd6ae89b5d331d278364d1848ed TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' [53950] upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:06:54 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53950/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53950/ Command failed (workunit test test_telemetry_squid.sh) on trial078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adc1923395f2bcd6ae89b5d331d278364d1848ed TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' [53952] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:10 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53952/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53952/ reached maximum tries (50) after waiting for 300 seconds [53953] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:07:53 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53953/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53953/ "2026-02-17T22:10:00.000276+0000 mon.a (mon.0) 909 : cluster [WRN] pg 4.b is stuck stale for 65s, current state stale+undersized+peered, last acting [0]" in cluster log [53954] upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:03:20 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53954/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53954/ "2026-02-17T22:50:00.000138+0000 mon.a (mon.0) 5949 : cluster [WRN] pg 1.5 is stuck peering for 2m, current state remapped+peering, last acting [2,6]" in cluster log [53955] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 02:21:03 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53955/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53955/ [53961] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:05:06 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53961/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53961/ "2026-02-17T23:30:00.000146+0000 mon.a (mon.0) 11670 : cluster [WRN] pg 1.a is stuck peering for 4m, current state remapped+peering, last acting [5,6,2]" in cluster log [53962] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:37 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53962/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53962/ reached maximum tries (50) after waiting for 300 seconds [53964] upgrade/squid-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:13:13 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53964/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53964/ Command failed (workunit test test_telemetry_squid.sh) on trial125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adc1923395f2bcd6ae89b5d331d278364d1848ed TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' [53971] upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:11:50 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53971/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53971/ "2026-02-17T22:14:14.597571+0000 mon.a (mon.0) 17 : cluster [WRN] pg 4.e is stuck peering for 66s, current state peering, last acting [4,6,2]" in cluster log [53974] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:03:24 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53974/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53974/ "2026-02-17T22:50:00.000128+0000 mon.a (mon.0) 2648 : cluster [WRN] daemon osd.1 on trial070 is in unknown state" in cluster log [53975] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:05:50 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53975/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53975/ "2026-02-17T22:30:00.000165+0000 mon.a (mon.0) 873 : cluster [WRN] pg 4.1 is stuck peering for 13m, current state peering, last acting [5,6,0]" in cluster log [53976] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:46 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53976/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53976/ reached maximum tries (50) after waiting for 300 seconds [53977] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/yes}} ----------------------------------------------------------------- time: 01:30:51 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53977/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53977/ [53978] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:01:29 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53978/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53978/ "2026-02-17T23:40:00.000162+0000 mon.a (mon.0) 13747 : cluster [WRN] daemon osd.0 on trial078 is in unknown state" in cluster log [53979] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 01:59:23 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53979/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53979/ "2026-02-17T22:40:59.193200+0000 mon.a (mon.0) 3255 : cluster [WRN] Health check failed: 1464/87326 objects unfound (1.676%) (OBJECT_UNFOUND)" in cluster log [53980] upgrade/telemetry-upgrade/reef-x/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks} ----------------------------------------------------------------- time: 00:12:32 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53980/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53980/ "2026-02-17T22:16:52.483233+0000 mon.a (mon.0) 165 : cluster [WRN] Health detail: HEALTH_WARN Telemetry requires re-opt-in" in cluster log [53981] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 01:31:13 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53981/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53981/ [53983] upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:06:27 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53983/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53983/ Command failed (workunit test test_telemetry_squid.sh) on trial085 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adc1923395f2bcd6ae89b5d331d278364d1848ed TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' [53984] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:53 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53984/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53984/ reached maximum tries (50) after waiting for 300 seconds [53987] upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 02:31:29 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53987/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53987/ "2026-02-17T23:20:00.000154+0000 mon.a (mon.0) 4573 : cluster [WRN] pg 1.12 is stuck peering for 4m, current state peering, last acting [5,1,6]" in cluster log [53988] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/reef 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}} ----------------------------------------------------------------- time: 00:15:59 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53988/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53988/ Command failed on trial080 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' [53989] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 02:17:19 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53989/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53989/ [54002] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 01:59:18 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54002/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54002/ "2026-02-17T23:10:00.000132+0000 mon.a (mon.0) 5111 : cluster [WRN] pg 1.13 is stuck peering for 11m, current state remapped+peering, last acting [3,4]" in cluster log [54004] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:19:04 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54004/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54004/ reached maximum tries (50) after waiting for 300 seconds [54008] upgrade/squid-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:06:44 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54008/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54008/ Command failed (workunit test test_telemetry_squid.sh) on trial144 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adc1923395f2bcd6ae89b5d331d278364d1848ed TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/test_telemetry_squid.sh' Dead ================================================================= [53970] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53970/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53970/ hit max job timeout [54016] upgrade/squid-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54016/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54016/ killed Pass ================================================================= [53929] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:10:08 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53929/ [53931] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:10:41 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53931/ [53939] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:22:52 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53939/ [53941] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/yes}} time: 01:28:48 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53941/ [53943] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:13:22 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53943/ [53945] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 01:51:04 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53945/ [53948] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/no}} time: 01:27:42 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53948/ [53949] upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:43:05 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53949/ [53951] upgrade/cephfs/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/reef 1-upgrade}} time: 00:05:06 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53951/ [53956] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:45:45 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53956/ [53957] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:39:27 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53957/ [53958] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:13:50 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53958/ [53959] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/yes}} time: 00:08:45 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53959/ [53960] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 01:57:44 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53960/ [53963] upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:41:52 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53963/ [53965] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/no}} time: 00:12:03 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53965/ [53966] upgrade/cephfs/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}} time: 00:04:17 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53966/ [53967] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:03:04 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53967/ [53968] upgrade/squid-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:03:00 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53968/ [53969] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:13:08 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53969/ [53972] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:10:49 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53972/ [53973] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}} time: 00:07:01 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53973/ [53982] upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:41:29 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53982/ [53985] upgrade/cephfs/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/reef 1-upgrade}} time: 00:04:29 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53985/ [53986] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:29:45 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53986/ [53990] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:13:50 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53990/ [53992] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:16:49 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53992/ [53994] upgrade/squid-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic overrides/ignorelist_health} time: 02:16:20 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53994/ [53996] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} time: 00:43:21 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53996/ [53998] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/yes}} time: 00:08:16 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/53998/ [54000] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity overrides/ignorelist_health} time: 02:06:28 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54000/ [54006] upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity overrides/ignorelist_health upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:41:56 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54006/ [54010] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/no}} time: 00:10:15 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54010/ [54012] upgrade/cephfs/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}} time: 00:04:18 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54012/ [54014] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity overrides/ignorelist_health} time: 02:13:27 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54014/ [54018] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:13:42 info: https://pulpito.ceph.com/yuriw-2026-02-17_21:13:22-upgrade-20.2.1_2.17.26-baseline_2-distro-default-trial/54018/ 2026-02-18T04:34:33.506 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/results.py", line 95, in email_results smtp = smtplib.SMTP('localhost') ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 255, in __init__ (code, msg) = self.connect(host, port) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 341, in connect self.sock = self._get_socket(host, port, self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket return socket.create_connection((host, port), timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection sock.connect(sa) File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect raise _SocketError(err, strerror(err)) ConnectionRefusedError: [Errno 111] Connection refused