2026-02-22T05:19:25.653 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-22T05:19:25.653 INFO:teuthology.results:subset: '111/120000'
2026-02-22T05:19:25.653 INFO:teuthology.results:seed: '6481'
2026-02-22T05:19:50.090 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-22T05:19:50.090 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2026-02-22T11:24:27.092 INFO:teuthology.results:Tests finished! gathering results...
2026-02-22T11:24:27.867 INFO:scrape:Found 26 jobs
2026-02-22T11:24:34.047 INFO:scrape:Found 15 distinct failure reasons
2026-02-22T11:24:34.047 INFO:scrape:Failure: Command failed on trial080 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2'
2026-02-22T11:24:34.047 INFO:scrape:2 jobs: ['63346', '63360']
2026-02-22T11:24:34.047 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}']
2026-02-22T11:24:34.047 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zstd}', 'random-objectstore$/{bluestore-hybrid}', 'tasks/e2e}']
2026-02-22T11:24:34.047 INFO:scrape:
2026-02-22T11:24:34.047 INFO:scrape:Failure: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on trial026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh'
2026-02-22T11:24:34.047 INFO:scrape:1 jobs: ['63362']
2026-02-22T11:24:34.047 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/osd}']
2026-02-22T11:24:34.047 INFO:scrape:
2026-02-22T11:24:34.047 INFO:scrape:Failure: Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest.test_list_enabled_module)
2026-02-22T11:24:34.047 INFO:scrape:2 jobs: ['63341', '63352']
2026-02-22T11:24:34.047 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/dashboard}']
2026-02-22T11:24:34.048 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'tasks/dashboard}']
2026-02-22T11:24:34.048 INFO:scrape:
2026-02-22T11:24:34.048 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on trial126 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2026-02-22T11:24:34.048 INFO:scrape:2 jobs: ['63340', '63363']
2026-02-22T11:24:34.048 INFO:scrape:suites intersection: ['agent/on', 'mon_election/connectivity', 'task/test_cephadm}']
2026-02-22T11:24:34.048 INFO:scrape:suites union: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}']
2026-02-22T11:24:34.048 INFO:scrape:
2026-02-22T11:24:34.048 INFO:scrape:Failure: 
2026-02-22T11:24:34.048 INFO:scrape:1 jobs: ['63351']
2026-02-22T11:24:34.048 INFO:scrape:suites: ['base', 'mem$/{normal-1}', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/{bluestore/{alloc$/{stupid}', 'onode-segment$/{512K}', 'rados', 'rados/singleton/{all/backfill-toofull', 'supported-random-distro$/{centos_latest}}', 'v1}}}}', 'write$/{v1/{compr$/{yes$/{snappy}}']
2026-02-22T11:24:34.048 INFO:scrape:
2026-02-22T11:24:34.048 INFO:scrape:Failure: Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on trial052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh'
2026-02-22T11:24:34.048 INFO:scrape:5 jobs: ['63338', '63348', '63354', '63345', '63361']
2026-02-22T11:24:34.048 INFO:scrape:suites intersection: ['base', 'rados', 'rados/singleton-bluestore/{all/cephtool']
2026-02-22T11:24:34.048 INFO:scrape:suites union: ['base', 'mem$/{low}', 'mem$/{normal-2}', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore/{alloc$/{btree}', 'objectstore/bluestore/{alloc$/{hybrid}', 'objectstore/bluestore/{alloc$/{stupid}', 'onode-segment$/{256K}', 'onode-segment$/{512K-onoff}', 'onode-segment$/{none}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}', 'v1}}}', 'v2}}}', 'write$/{v1/{compr$/{no$/{no}}', 'write$/{v2/{compr$/{no$/{no}}', 'write$/{v2/{compr$/{yes$/{zlib}}']
2026-02-22T11:24:34.048 INFO:scrape:
2026-02-22T11:24:34.048 INFO:scrape:Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial053 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh'
2026-02-22T11:24:34.048 INFO:scrape:ceph version 20.3.0-5397-gf7273f30 (f7273f3049146bdd7fa7b4a400114872a50228e9) tentacle (dev - RelWithDebInfo)
 1: /lib64/libc.so.6(+0x3fc30) [0x7f34a0e3fc30]
 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xc7) [0x7d9c67]
 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, KeyValueDB::IteratorBounds)+0x2d5) [0x7d49e5]
 4: (BlueStore::_close_db()+0x44a) [0x6291ba]
 5: (BlueStore::umount()+0x1ac) [0x632b2c]
 6: ceph-kvstore-tool() [0x5ce063]
 7: main()
 8: /lib64/libc.so.6(+0x2a610) [0x7f34a0e2a610]
 9: __libc_start_main()
 10: _start()
2026-02-22T11:24:34.048 INFO:scrape:2 jobs: ['63343', '63355']
2026-02-22T11:24:34.048 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_latest}}']
2026-02-22T11:24:34.048 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_latest}}']
2026-02-22T11:24:34.048 INFO:scrape:
2026-02-22T11:24:34.048 INFO:scrape:Failure: Command failed on trial045 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:f7273f3049146bdd7fa7b4a400114872a50228e9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4ed37069-0faf-11f1-8cfc-d404e6e7d460 -- bash -c \'set -ex\n# since we don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in order to replace the fake hostnames "host.a/b/c" with\n# the actual names cephadm knows the host by within the mon spec\nceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'a\\nb\\nc\'"\'"\' > mon_ids\necho $\'"\'"\'{datacenter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' > crush_locs\nceph orch ls --service-name mon --export > mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3 && read fakename <&4; do\n  MONSPEC="${MONSPEC//$fakename/$realname}"\ndone 3<realnames 4<fakenames\necho "$MONSPEC" > mon.yaml\ncat mon.yaml\n# now the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep 90\nceph mon dump\nceph mon dump --format json\n# verify all the crush locations got set from "ceph mon dump" output\nwhile read monid <&3 && read crushloc <&4; do\n  ceph mon dump --format json | jq --arg monid "$monid" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] | select(.name == $monid) | .crush_location == $crushloc\'"\'"\'\ndone 3<mon_ids 4<crush_locs\n\''
2026-02-22T11:24:34.048 INFO:scrape:1 jobs: ['63359']
2026-02-22T11:24:34.048 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_set_mon_crush_locations}']
2026-02-22T11:24:34.048 INFO:scrape:
2026-02-22T11:24:34.048 INFO:scrape:Failure: Test failure: test_standby (tasks.mgr.test_prometheus.TestPrometheus.test_standby)
2026-02-22T11:24:34.048 INFO:scrape:1 jobs: ['63357']
2026-02-22T11:24:34.049 INFO:scrape:suites: ['2-ceph', '3-mgrmodules', '4-units/prometheus}}', 'base', 'debug/mgr', 'distro/{centos_latest}', 'mem$/{normal-2}', 'mgr_ttl_cache/disable', 'mon_election/classic', 'onode-segment$/{256K}', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore/{alloc$/{avl}', 'tasks/{1-install', 'v1}}}}', 'write$/{v1/{compr$/{no$/{no}}']
2026-02-22T11:24:34.049 INFO:scrape:
2026-02-22T11:24:34.049 INFO:scrape:Dead: b'2026-02-22T11:23:42.112 DEBUG:teuthology.exit:Finished running handlers'
2026-02-22T11:24:34.049 INFO:scrape:1 jobs: ['63347']
2026-02-22T11:24:34.049 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_iscsi_container/{centos_9.stream', 'test_iscsi_container}}']
2026-02-22T11:24:34.049 INFO:scrape:
2026-02-22T11:24:34.049 INFO:scrape:Failure: Command failed (workunit test post-file.sh) on trial119 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh'
2026-02-22T11:24:34.049 INFO:scrape:2 jobs: ['63344', '63356']
2026-02-22T11:24:34.049 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}']
2026-02-22T11:24:34.049 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}']
2026-02-22T11:24:34.049 INFO:scrape:
2026-02-22T11:24:34.049 INFO:scrape:Timeout 3h running clone.client.0/qa/standalone/misc/test-ceph-helpers.sh
2026-02-22T11:24:34.049 INFO:scrape:1 jobs: ['63349']
2026-02-22T11:24:34.049 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/misc}']
2026-02-22T11:24:34.049 INFO:scrape:
2026-02-22T11:24:34.049 INFO:scrape:Failure: Command failed (workunit test mon/misc.sh) on trial116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh'
2026-02-22T11:24:34.049 INFO:scrape:1 jobs: ['63358']
2026-02-22T11:24:34.049 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/mon}']
2026-02-22T11:24:34.049 INFO:scrape:
2026-02-22T11:24:34.049 INFO:scrape:Failure: Command failed (workunit test dencoder/test-dencoder.sh) on trial176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/dencoder/test-dencoder.sh'
2026-02-22T11:24:34.049 INFO:scrape:1 jobs: ['63342']
2026-02-22T11:24:34.049 INFO:scrape:suites: ['1-tasks', 'rados/encoder/{0-start', 'supported-random-distro$/{centos_latest}}']
2026-02-22T11:24:34.049 INFO:scrape:
2026-02-22T11:24:34.049 INFO:scrape:Failure: Command failed on trial055 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:f7273f3049146bdd7fa7b4a400114872a50228e9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 549a7e74-0faf-11f1-bffc-d404e6e7d460 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n  curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v2/status\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts | jq -e \'"\'"\'.[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n# check prometheus metrics endpoint is not empty and make sure we can get metrics\nMETRICS_URL=$(ceph mgr services | jq -r .prometheus)\n[ -n "$METRICS_URL" ] || exit 1\ncurl -s "${METRICS_URL}metrics" | grep -q \'"\'"\'^ceph_health_status\'"\'"\'\n\''
2026-02-22T11:24:34.049 INFO:scrape:1 jobs: ['63350']
2026-02-22T11:24:34.049 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_monitoring_stack_basic}']
2026-02-22T11:24:34.049 INFO:scrape:
2026-02-22T11:24:34.050 INFO:teuthology.results:Sending results to skanta@redhat.com: Test Run: skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial
=================================================================
info:    https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/
logs:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/
failed:  23
dead:    1
running: 0
waiting: 0
queued:  0
passed:  2


Fail
=================================================================
[63338]  rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore/{alloc$/{stupid} base mem$/{normal-2} onode-segment$/{none} write$/{v2/{compr$/{no$/{no}} v2}}} rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:22:51
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63338/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63338/

    Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on
    trial052 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh'

[63340]  rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:11:00
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63340/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63340/

    Command failed (workunit test cephadm/test_cephadm.sh) on trial126 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[63341]  rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-lz4} tasks/dashboard}
-----------------------------------------------------------------
time:   00:22:39
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63341/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63341/

    Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module
    .MgrModuleTest.test_list_enabled_module)

[63342]  rados/encoder/{0-start 1-tasks supported-random-distro$/{centos_latest}}
-----------------------------------------------------------------
time:   00:06:40
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63342/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63342/

    Command failed (workunit test dencoder/test-dencoder.sh) on trial176 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/dencoder/test-
    dencoder.sh'

[63343]  rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_latest}}
-----------------------------------------------------------------
time:   00:05:30
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63343/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63343/

    Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial053
    with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh'

[63344]  rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:05:38
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63344/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63344/

    Command failed (workunit test post-file.sh) on trial119 with status 255:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh'

[63345]  rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore/{alloc$/{btree} base mem$/{low} onode-segment$/{512K-onoff} write$/{v2/{compr$/{no$/{no}} v2}}} rados supported-random-distro$/{centos_latest}}
-----------------------------------------------------------------
time:   00:23:29
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63345/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63345/

    Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on
    trial050 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh'

[63346]  rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/e2e}
-----------------------------------------------------------------
time:   00:05:52
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63346/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63346/

    Command failed on trial080 with status 1: 'yes | sudo mkfs.xfs -f -i
    size=2048 /dev/vg_nvme/lv_2'

[63348]  rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore/{alloc$/{stupid} base mem$/{low} onode-segment$/{256K} write$/{v2/{compr$/{yes$/{zlib}} v2}}} rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:24:23
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63348/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63348/

    Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on
    trial149 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh'

[63349]  rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc}
-----------------------------------------------------------------
time:   03:15:11
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63349/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63349/

    Command failed (workunit test misc/test-ceph-helpers.sh) on trial185 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-
    helpers.sh'

[63350]  rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic}
-----------------------------------------------------------------
time:   00:17:03
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63350/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63350/

    Command failed on trial055 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:f7273f3049146bdd7fa7b4a400114872a50228e9 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    549a7e74-0faf-11f1-bffc-d404e6e7d460 -- bash -c \'set -e\nset -x\nceph orch
    apply node-exporter\nceph orch apply grafana\nceph orch apply
    alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph
    orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f
    json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch
    ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r
    \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type
    prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r
    \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type
    alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r
    \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r
    --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] |
    select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host
    ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] |
    select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host
    ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] |
    select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-
    exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls
    -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n
    curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are
    responsive and database health is okay\ncurl -k -s
    https://${GRAFANA_IP}:3000/api/health\ncurl -k -s
    https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database ==
    "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch
    daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are
    responsive and mon down alert is firing\ncurl -s
    http://${PROM_IP}:9095/api/v1/status/config\ncurl -s
    http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status ==
    "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s
    http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[]
    | select(.labels | .alertname == "CephMonDown") | .state ==
    "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon
    down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v2/status\ncurl
    -s http://${ALERTM_IP}:9093/api/v2/alerts\ncurl -s
    http://${ALERTM_IP}:9093/api/v2/alerts | jq -e \'"\'"\'.[] | select(.labels
    | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n#
    check prometheus metrics endpoint is not empty and make sure we can get
    metrics\nMETRICS_URL=$(ceph mgr services | jq -r .prometheus)\n[ -n
    "$METRICS_URL" ] || exit 1\ncurl -s "${METRICS_URL}metrics" | grep -q
    \'"\'"\'^ceph_health_status\'"\'"\'\n\''

[63351]  rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{stupid} base mem$/{normal-1} onode-segment$/{512K} write$/{v1/{compr$/{yes$/{snappy}} v1}}}} rados supported-random-distro$/{centos_latest}}
-----------------------------------------------------------------
time:   00:30:52
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63351/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63351/

[63352]  rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} tasks/dashboard}
-----------------------------------------------------------------
time:   00:22:56
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63352/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63352/

    Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module
    .MgrModuleTest.test_list_enabled_module)

[63354]  rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore/{alloc$/{hybrid} base mem$/{low} onode-segment$/{512K-onoff} write$/{v1/{compr$/{no$/{no}} v1}}} rados supported-random-distro$/{centos_latest}}
-----------------------------------------------------------------
time:   00:23:41
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63354/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63354/

    Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on
    trial194 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh'

[63355]  rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_latest}}
-----------------------------------------------------------------
time:   00:05:41
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63355/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63355/

    Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial133
    with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh'

[63356]  rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:05:38
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63356/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63356/

    Command failed (workunit test post-file.sh) on trial077 with status 255:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh'

[63357]  rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore/{alloc$/{avl} base mem$/{normal-2} onode-segment$/{256K} write$/{v1/{compr$/{no$/{no}} v1}}}} tasks/{1-install 2-ceph 3-mgrmodules 4-units/prometheus}}
-----------------------------------------------------------------
time:   00:06:06
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63357/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63357/

    Test failure: test_standby
    (tasks.mgr.test_prometheus.TestPrometheus.test_standby)

[63358]  rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon}
-----------------------------------------------------------------
time:   00:09:10
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63358/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63358/

    Command failed (workunit test mon/misc.sh) on trial116 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh'

[63359]  rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations}
-----------------------------------------------------------------
time:   00:09:40
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63359/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63359/

    Command failed on trial045 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:f7273f3049146bdd7fa7b4a400114872a50228e9 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4ed37069-0faf-11f1-8cfc-d404e6e7d460 -- bash -c \'set -ex\n# since we
    don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in
    order to replace the fake hostnames "host.a/b/c" with\n# the actual names
    cephadm knows the host by within the mon spec\nceph orch host ls --format
    json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho
    $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho
    $\'"\'"\'a\\nb\\nc\'"\'"\' > mon_ids\necho $\'"\'"\'{datacenter=a}\\n{datac
    enter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' > crush_locs\nceph orch ls
    --service-name mon --export > mon.yaml\nMONSPEC=`cat mon.yaml`\necho
    "$MONSPEC"\nwhile read realname <&3 && read fakename <&4; do\n
    MONSPEC="${MONSPEC//$fakename/$realname}"\ndone 3<realnames
    4<fakenames\necho "$MONSPEC" > mon.yaml\ncat mon.yaml\n# now the spec
    should have the real hostnames, so let\'"\'"\'s re-apply\nceph orch apply
    -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls --service-name
    mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep 90\nceph mon
    dump\nceph mon dump --format json\n# verify all the crush locations got set
    from "ceph mon dump" output\nwhile read monid <&3 && read crushloc <&4;
    do\n  ceph mon dump --format json | jq --arg monid "$monid" --arg crushloc
    "$crushloc" -e \'"\'"\'.mons | .[] | select(.name == $monid) |
    .crush_location == $crushloc\'"\'"\'\ndone 3<mon_ids 4<crush_locs\n\''

[63360]  rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/e2e}
-----------------------------------------------------------------
time:   00:05:49
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63360/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63360/

    Command failed on trial071 with status 1: 'yes | sudo mkfs.xfs -f -i
    size=2048 /dev/vg_nvme/lv_2'

[63361]  rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore/{alloc$/{stupid} base mem$/{normal-2} onode-segment$/{none} write$/{v1/{compr$/{no$/{no}} v1}}} rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:24:05
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63361/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63361/

    Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on
    trial040 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh'

[63362]  rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd}
-----------------------------------------------------------------
time:   00:27:04
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63362/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63362/

    Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on trial026
    with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd
    -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-
    ops.sh'

[63363]  rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:10:02
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63363/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63363/

    Command failed (workunit test cephadm/test_cephadm.sh) on trial092 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'



Dead
=================================================================
[63347]  rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63347/
log:    https://qa-proxy.ceph.com/teuthology/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63347/

    killed



Pass
=================================================================
[63339] rados/basic/{ceph clusters/{fixed-2} mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore/{alloc$/{bitmap} base mem$/{normal-2} onode-segment$/{512K-onoff} write$/{random/{compr$/{yes$/{snappy}} random}}}} rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite}
time:   00:09:45
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63339/

[63353] rados/thrash-erasure-code/{ceph clusters/{fixed-4} ec_optimizations/ec_optimizations_on fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore/{alloc$/{bitmap} base mem$/{normal-1} onode-segment$/{512K-onoff} write$/{v2/{compr$/{no$/{no}} v2}}}} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
time:   00:21:44
info:   https://pulpito.ceph.com/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63353/
2026-02-22T11:24:34.057 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 33, in main
    results(args['--archive-dir'], args['--name'], args['--email'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 78, in results
    email_results(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 95, in email_results
    smtp = smtplib.SMTP('localhost')
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 341, in connect
    self.sock = self._get_socket(host, port, self.timeout)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket
    return socket.create_connection((host, port), timeout,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection
    sock.connect(sa)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect
    self._internal_connect(address)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect
    raise _SocketError(err, strerror(err))
ConnectionRefusedError: [Errno 111] Connection refused