2026-02-12T16:52:49.002 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71 2026-02-12T16:52:49.002 INFO:teuthology.results:subset: '111/120000' 2026-02-12T16:52:49.002 INFO:teuthology.results:seed: '7172' 2026-02-12T20:28:54.832 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71 2026-02-12T20:28:54.840 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-13T08:29:45.742 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2026-02-13T08:29:51.554 INFO:scrape:Found 272 jobs 2026-02-13T08:29:52.202 WARNING:scrape:Missing teuthology log /home/teuthworker/mnt/teuthology/yuriw-2026-02-12_15:38:14-rados-wip-yuri4-testing-2026-02-11-2126-distro-default-trial/46483/teuthology.log 2026-02-13T08:29:58.653 INFO:scrape:Found 23 distinct failure reasons 2026-02-13T08:29:58.653 INFO:scrape:Failure: wait_for_recovery: failed before timeout expired 2026-02-13T08:29:58.653 INFO:scrape:2 jobs: ['46405', '46548'] 2026-02-13T08:29:58.654 INFO:scrape:suites intersection: ['base', 'mem$/{low}', 'mon_election/connectivity', 'objectstore/{bluestore/{alloc$/{bitmap}', 'rados', 'thrashosds-health'] 2026-02-13T08:29:58.654 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'base', 'ceph', 'cluster/{12-osds}', 'clusters/{fixed-4}', 'crc-failures/default', 'd-balancer/crush-compat', 'ec_optimizations/ec_optimizations_off', 'mem$/{low}', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/{bluestore/{alloc$/{bitmap}', 'onode-segment$/{256K}', 'onode-segment$/{512K-onoff}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'random}}}}', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'v1}}}}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/pool-snaps-few-objects}', 'write$/{random/{compr$/{yes$/{zstd}}', 'write$/{v1/{compr$/{yes$/{lz4}}'] 2026-02-13T08:29:58.654 INFO:scrape: 2026-02-13T08:29:58.654 INFO:scrape:Failure: 2026-02-13T08:29:58.654 INFO:scrape:2 jobs: ['46441', '46634'] 2026-02-13T08:29:58.654 INFO:scrape:suites intersection: ['base', 'mon_election/classic', 'rados'] 2026-02-13T08:29:58.654 INFO:scrape:suites union: ['base', 'mem$/{normal-1}', 'mem$/{normal-2}', 'mon_election/classic', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v2only', 'objectstore/{bluestore/{alloc$/{bitmap}', 'objectstore/{bluestore/{alloc$/{btree}', 'onode-segment$/{1M}', 'onode-segment$/{512K}', 'rados', 'rados/singleton/{all/backfill-toofull', 'rados/singleton/{all/osd-recovery-incomplete', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}', 'v1}}}}', 'v2}}}}', 'write$/{v1/{compr$/{no$/{no}}', 'write$/{v2/{compr$/{no$/{no}}'] 2026-02-13T08:29:58.654 INFO:scrape: 2026-02-13T08:29:58.654 INFO:scrape:Failure: Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on trial168 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afefaa40503a4bf024e96f97dd7f852b75f00ad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh' 2026-02-13T08:29:58.654 INFO:scrape:4 jobs: ['46714', '46482', '46588', '46653'] 2026-02-13T08:29:58.654 INFO:scrape:suites intersection: ['base', 'rados', 'rados/singleton-bluestore/{all/cephtool'] 2026-02-13T08:29:58.654 INFO:scrape:suites union: ['base', 'mem$/{low}', 'mem$/{normal-2}', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore/{alloc$/{avl}', 'objectstore/bluestore/{alloc$/{bitmap}', 'onode-segment$/{1M}', 'onode-segment$/{512K-onoff}', 'onode-segment$/{512K}', 'onode-segment$/{none}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'random}}}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}', 'v2}}}', 'write$/{random/{compr$/{no$/{no}}', 'write$/{v2/{compr$/{no$/{no}}'] 2026-02-13T08:29:58.654 INFO:scrape: 2026-02-13T08:29:58.654 INFO:scrape:Failure: "grep: /var/log/ceph/e576c4ab-084c-11f1-a729-d404e6e7d460/ceph.log: No such file or directory" in cluster log 2026-02-13T08:29:58.654 INFO:scrape:2 jobs: ['46640', '46435'] 2026-02-13T08:29:58.654 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'overrides/ignorelist_health', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2026-02-13T08:29:58.654 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'overrides/ignorelist_health', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc}', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2026-02-13T08:29:58.654 INFO:scrape: 2026-02-13T08:29:58.654 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on trial080 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afefaa40503a4bf024e96f97dd7f852b75f00ad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2026-02-13T08:29:58.654 INFO:scrape:1 jobs: ['46400'] 2026-02-13T08:29:58.654 INFO:scrape:suites: ['base', 'mem$/{low}', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore/{alloc$/{hybrid}', 'onode-segment$/{none}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}', 'v1}}}', 'write$/{v1/{compr$/{yes$/{zstd}}'] 2026-02-13T08:29:58.654 INFO:scrape: 2026-02-13T08:29:58.654 INFO:scrape:Dead: None 2026-02-13T08:29:58.654 INFO:scrape:4 jobs: ['46483', '46568', '46691', '46712'] 2026-02-13T08:29:58.654 INFO:scrape: 2026-02-13T08:29:58.654 INFO:scrape:Failure: Command failed (workunit test post-file.sh) on trial165 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afefaa40503a4bf024e96f97dd7f852b75f00ad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh' 2026-02-13T08:29:58.655 INFO:scrape:2 jobs: ['46463', '46663'] 2026-02-13T08:29:58.655 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}'] 2026-02-13T08:29:58.655 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}'] 2026-02-13T08:29:58.655 INFO:scrape: 2026-02-13T08:29:58.655 INFO:scrape:Failure: Test failure: test_standby (tasks.mgr.test_prometheus.TestPrometheus.test_standby) 2026-02-13T08:29:58.655 INFO:scrape:1 jobs: ['46665'] 2026-02-13T08:29:58.655 INFO:scrape:suites: ['2-ceph', '3-mgrmodules', '4-units/prometheus}}', 'base', 'debug/mgr', 'distro/{centos_latest}', 'mem$/{low}', 'mgr_ttl_cache/disable', 'mon_election/classic', 'onode-segment$/{256K}', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore/{alloc$/{hybrid}', 'tasks/{1-install', 'v1}}}}', 'write$/{v1/{compr$/{yes$/{zlib}}'] 2026-02-13T08:29:58.655 INFO:scrape: 2026-02-13T08:29:58.655 INFO:scrape:Failure: Failed to reconnect to trial104 2026-02-13T08:29:58.655 INFO:scrape:1 jobs: ['46420'] 2026-02-13T08:29:58.655 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'base', 'ceph', 'clusters/{fixed-4}', 'crc-failures/default', 'd-balancer/read', 'mem$/{low}', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/{bluestore/{alloc$/{btree}', 'onode-segment$/{1M}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'random}}}}', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects-localized}', 'write$/{random/{compr$/{no$/{no}}'] 2026-02-13T08:29:58.655 INFO:scrape: 2026-02-13T08:29:58.655 INFO:scrape:Failure: Command failed on trial027 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2' 2026-02-13T08:29:58.655 INFO:scrape:2 jobs: ['46542', '46705'] 2026-02-13T08:29:58.655 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}'] 2026-02-13T08:29:58.655 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zstd}', 'random-objectstore$/{bluestore-stupid}', 'tasks/e2e}'] 2026-02-13T08:29:58.655 INFO:scrape: 2026-02-13T08:29:58.655 INFO:scrape:Failure: Command failed (workunit test mon/misc.sh) on trial050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afefaa40503a4bf024e96f97dd7f852b75f00ad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh' 2026-02-13T08:29:58.655 INFO:scrape:1 jobs: ['46667'] 2026-02-13T08:29:58.655 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] 2026-02-13T08:29:58.655 INFO:scrape: 2026-02-13T08:29:58.655 INFO:scrape:Failure: Command failed (workunit test osd-backfill/osd-backfill-prio.sh) on trial020 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afefaa40503a4bf024e96f97dd7f852b75f00ad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh' 2026-02-13T08:29:58.655 INFO:scrape:1 jobs: ['46694'] 2026-02-13T08:29:58.655 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/osd-backfill}'] 2026-02-13T08:29:58.655 INFO:scrape: 2026-02-13T08:29:58.655 INFO:scrape:Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial110 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afefaa40503a4bf024e96f97dd7f852b75f00ad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' 2026-02-13T08:29:58.655 INFO:scrape:ceph version 20.3.0-5230-gafefaa40 (afefaa40503a4bf024e96f97dd7f852b75f00ad6) tentacle (dev - RelWithDebInfo) 1: /lib64/libc.so.6(+0x3fc30) [0x7f139423fc30] 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xc7) [0x7d9747] 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x2d5) [0x7d44c5] 4: (BlueStore::_close_db()+0x44a) [0x62931a] 5: (BlueStore::umount()+0x1ac) [0x632c4c] 6: ceph-kvstore-tool() [0x5cdf13] 7: main() 8: /lib64/libc.so.6(+0x2a610) [0x7f139422a610] 9: __libc_start_main() 10: _start() 2026-02-13T08:29:58.655 INFO:scrape:2 jobs: ['46452', '46657'] 2026-02-13T08:29:58.655 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_latest}}'] 2026-02-13T08:29:58.655 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_latest}}'] 2026-02-13T08:29:58.656 INFO:scrape: 2026-02-13T08:29:58.656 INFO:scrape:Failure: Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest.test_list_enabled_module) 2026-02-13T08:29:58.656 INFO:scrape:2 jobs: ['46636', '46428'] 2026-02-13T08:29:58.656 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/dashboard}'] 2026-02-13T08:29:58.656 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-snappy}', 'random-objectstore$/{bluestore-stupid}', 'tasks/dashboard}'] 2026-02-13T08:29:58.656 INFO:scrape: 2026-02-13T08:29:58.656 INFO:scrape:Failure: Command failed on trial059 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:afefaa40503a4bf024e96f97dd7f852b75f00ad6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cabc3ff0-0848-11f1-99da-d404e6e7d460 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v2/status\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts | jq -e \'"\'"\'.[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n# check prometheus metrics endpoint is not empty and make sure we can get metrics\nMETRICS_URL=$(ceph mgr services | jq -r .prometheus)\n[ -n "$METRICS_URL" ] || exit 1\ncurl -s "${METRICS_URL}metrics" | grep -q \'"\'"\'^ceph_health_status\'"\'"\'\n\'' 2026-02-13T08:29:58.656 INFO:scrape:1 jobs: ['46622'] 2026-02-13T08:29:58.656 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_monitoring_stack_basic}'] 2026-02-13T08:29:58.656 INFO:scrape: 2026-02-13T08:29:58.656 INFO:scrape:Timeout 3h running clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2026-02-13T08:29:58.656 INFO:scrape:1 jobs: ['46611'] 2026-02-13T08:29:58.656 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/misc}'] 2026-02-13T08:29:58.656 INFO:scrape: 2026-02-13T08:29:58.656 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on trial132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afefaa40503a4bf024e96f97dd7f852b75f00ad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2026-02-13T08:29:58.656 INFO:scrape:2 jobs: ['46723', '46418'] 2026-02-13T08:29:58.656 INFO:scrape:suites intersection: ['agent/on', 'mon_election/connectivity', 'task/test_cephadm}'] 2026-02-13T08:29:58.656 INFO:scrape:suites union: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] 2026-02-13T08:29:58.656 INFO:scrape: 2026-02-13T08:29:58.656 INFO:scrape:Failure: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on trial099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afefaa40503a4bf024e96f97dd7f852b75f00ad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2026-02-13T08:29:58.656 INFO:scrape:1 jobs: ['46720'] 2026-02-13T08:29:58.656 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/osd}'] 2026-02-13T08:29:58.656 INFO:scrape: 2026-02-13T08:29:58.656 INFO:scrape:Failure: Command failed (workunit test rados/test_librados_build.sh) on trial047 with status 6: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afefaa40503a4bf024e96f97dd7f852b75f00ad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_librados_build.sh' 2026-02-13T08:29:58.656 INFO:scrape:1 jobs: ['46559'] 2026-02-13T08:29:58.656 INFO:scrape:suites: ['mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/librados_hello_world', 'supported-random-distro$/{centos_latest}}'] 2026-02-13T08:29:58.656 INFO:scrape: 2026-02-13T08:29:58.656 INFO:scrape:Failure: Command failed on trial063 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:afefaa40503a4bf024e96f97dd7f852b75f00ad6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 350e643c-0850-11f1-9486-d404e6e7d460 -- bash -c \'set -ex\n# since we don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in order to replace the fake hostnames "host.a/b/c" with\n# the actual names cephadm knows the host by within the mon spec\nceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'a\\nb\\nc\'"\'"\' > mon_ids\necho $\'"\'"\'{datacenter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' > crush_locs\nceph orch ls --service-name mon --export > mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3 && read fakename <&4; do\n MONSPEC="${MONSPEC//$fakename/$realname}"\ndone 3 mon.yaml\ncat mon.yaml\n# now the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep 90\nceph mon dump\nceph mon dump --format json\n# verify all the crush locations got set from "ceph mon dump" output\nwhile read monid <&3 && read crushloc <&4; do\n ceph mon dump --format json | jq --arg monid "$monid" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] | select(.name == $monid) | .crush_location == $crushloc\'"\'"\'\ndone 3 realnames\necho $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'a\\nb\\nc\'"\'"\' > mon_ids\necho $\'"\'"\'{datacenter=a}\\n{datac enter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' > crush_locs\nceph orch ls --service-name mon --export > mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3 && read fakename <&4; do\n MONSPEC="${MONSPEC//$fakename/$realname}"\ndone 3 mon.yaml\ncat mon.yaml\n# now the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep 90\nceph mon dump\nceph mon dump --format json\n# verify all the crush locations got set from "ceph mon dump" output\nwhile read monid <&3 && read crushloc <&4; do\n ceph mon dump --format json | jq --arg monid "$monid" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] | select(.name == $monid) | .crush_location == $crushloc\'"\'"\'\ndone 3