2026-02-17T20:45:02.576 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71
2026-02-17T20:45:02.576 INFO:teuthology.results:subset: '111/120000'
2026-02-17T20:45:02.576 INFO:teuthology.results:seed: '2987'
2026-02-17T21:09:51.629 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71
2026-02-17T21:09:51.636 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2026-02-18T09:10:02.277 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2026-02-18T09:10:03.516 INFO:scrape:Found 57 jobs
2026-02-18T09:10:03.519 WARNING:scrape:Missing teuthology log /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_20:43:43-rados-wip-yuri6-testing-2026-02-17-1732-squid-distro-default-trial/53858/teuthology.log
2026-02-18T09:10:10.881 INFO:scrape:Found 13 distinct failure reasons
2026-02-18T09:10:10.881 INFO:scrape:Failure: "2026-02-17T21:19:52.722736+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 97991.612898 IOPS is not within the threshold limit range of 1000.000000 IOPS and 80000.000000 IOPS for osd.0. IOPS capacity is unchanged at 21500.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2026-02-18T09:10:10.881 INFO:scrape:8 jobs: ['53867', '53882', '53891', '53887', '53876', '53909', '53901', '53859']
2026-02-18T09:10:10.881 INFO:scrape:suites intersection: []
2026-02-18T09:10:10.881 INFO:scrape:suites union: ['0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '0-start', '1-install/quincy', '1-install/reef', '1-tasks', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'overrides/ignorelist_health', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_9.stream}', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}']
2026-02-18T09:10:10.882 INFO:scrape:
2026-02-18T09:10:10.882 INFO:scrape:Dead: None
2026-02-18T09:10:10.882 INFO:scrape:1 jobs: ['53858']
2026-02-18T09:10:10.882 INFO:scrape:
2026-02-18T09:10:10.882 INFO:scrape:Failure: "2026-02-17T21:25:19.886518+0000 mon.a (mon.0) 165 : cluster [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT)" in cluster log
2026-02-18T09:10:10.882 INFO:scrape:9 jobs: ['53902', '53880', '53895', '53862', '53886', '53874', '53878', '53894', '53897']
2026-02-18T09:10:10.882 INFO:scrape:suites intersection: []
2026-02-18T09:10:10.882 INFO:scrape:suites union: ['1-start', '2-ceph', '2-inject-leak/none', '2-inject-leak/osd', '2-services/rgw', '3-final}', '3-mgrmodules', '4-units/module_selftest}}', 'centos_latest}', 'ceph', 'clusters/fixed-2', 'clusters/fixed-4', 'd-thrash/default/{default', 'd-thrash/none', 'debug/mgr', 'distro/{centos_latest}', 'mgr_ttl_cache/enable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream}', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'random-objectstore$/{bluestore-comp-snappy}', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'tasks/{1-install', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}']
2026-02-18T09:10:10.882 INFO:scrape:
2026-02-18T09:10:10.882 INFO:scrape:Failure: Command failed (workunit test post-file.sh) on trial007 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d855f53b89fdcec760fd9232a5fb55ed4fb111a1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh'
2026-02-18T09:10:10.882 INFO:scrape:2 jobs: ['53892', '53866']
2026-02-18T09:10:10.882 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file']
2026-02-18T09:10:10.882 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}']
2026-02-18T09:10:10.882 INFO:scrape:
2026-02-18T09:10:10.882 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on trial071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d855f53b89fdcec760fd9232a5fb55ed4fb111a1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
2026-02-18T09:10:10.882 INFO:scrape:4 jobs: ['53870', '53884', '53900', '53856']
2026-02-18T09:10:10.882 INFO:scrape:suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool']
2026-02-18T09:10:10.882 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}']
2026-02-18T09:10:10.882 INFO:scrape:
2026-02-18T09:10:10.882 INFO:scrape:Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial060 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d855f53b89fdcec760fd9232a5fb55ed4fb111a1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh'
2026-02-18T09:10:10.882 INFO:scrape:ceph version 19.2.3-679-gd855f53b (d855f53b89fdcec760fd9232a5fb55ed4fb111a1) squid (stable)
 1: /lib64/libc.so.6(+0x3fc30) [0x7f0212e3fc30]
 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x563f1185ee54]
 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, KeyValueDB::IteratorBounds)+0x240) [0x563f1185f570]
 4: (BlueStore::_close_db()+0x380) [0x563f1169c470]
 5: (BlueStore::umount()+0x244) [0x563f116a8fe4]
 6: ceph-kvstore-tool(+0x1f7503) [0x563f11646503]
 7: main()
 8: /lib64/libc.so.6(+0x2a610) [0x7f0212e2a610]
 9: __libc_start_main()
 10: _start()
2026-02-18T09:10:10.882 INFO:scrape:2 jobs: ['53863', '53889']
2026-02-18T09:10:10.882 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_latest}}']
2026-02-18T09:10:10.882 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_latest}}']
2026-02-18T09:10:10.882 INFO:scrape:
2026-02-18T09:10:10.883 INFO:scrape:Failure: Command failed on trial127 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d855f53b89fdcec760fd9232a5fb55ed4fb111a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6955d983-0c44-11f1-b9a6-d404e6e7d460 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n  curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v2/status\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts | jq -e \'"\'"\'.[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n\''
2026-02-18T09:10:10.883 INFO:scrape:1 jobs: ['53883']
2026-02-18T09:10:10.883 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_monitoring_stack_basic}']
2026-02-18T09:10:10.883 INFO:scrape:
2026-02-18T09:10:10.883 INFO:scrape:Failure: Command failed on trial170 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_2'
2026-02-18T09:10:10.883 INFO:scrape:2 jobs: ['53873', '53905']
2026-02-18T09:10:10.883 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}']
2026-02-18T09:10:10.883 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-bitmap}', 'random-objectstore$/{bluestore-comp-zstd}', 'tasks/e2e}']
2026-02-18T09:10:10.883 INFO:scrape:
2026-02-18T09:10:10.883 INFO:scrape:Failure: 
2026-02-18T09:10:10.883 INFO:scrape:1 jobs: ['53904']
2026-02-18T09:10:10.883 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/backfill-toofull', 'supported-random-distro$/{centos_latest}}']
2026-02-18T09:10:10.883 INFO:scrape:
2026-02-18T09:10:10.883 INFO:scrape:Failure: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on trial031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d855f53b89fdcec760fd9232a5fb55ed4fb111a1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh'
2026-02-18T09:10:10.883 INFO:scrape:1 jobs: ['53907']
2026-02-18T09:10:10.883 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd}']
2026-02-18T09:10:10.883 INFO:scrape:
2026-02-18T09:10:10.883 INFO:scrape:Failure: Command failed (workunit test osd-backfill/osd-backfill-prio.sh) on trial091 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d855f53b89fdcec760fd9232a5fb55ed4fb111a1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh'
2026-02-18T09:10:10.883 INFO:scrape:1 jobs: ['53899']
2026-02-18T09:10:10.883 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd-backfill}']
2026-02-18T09:10:10.883 INFO:scrape:
2026-02-18T09:10:10.883 INFO:scrape:Failure: Failed to reconnect to trial013
2026-02-18T09:10:10.883 INFO:scrape:1 jobs: ['53860']
2026-02-18T09:10:10.883 INFO:scrape:suites: ['2-inject-leak/mon', 'centos_latest}', 'rados/valgrind-leaks/{1-start']
2026-02-18T09:10:10.883 INFO:scrape:
2026-02-18T09:10:10.883 INFO:scrape:Failure: Test failure: test_standby (tasks.mgr.test_prometheus.TestPrometheus.test_standby)
2026-02-18T09:10:10.883 INFO:scrape:1 jobs: ['53893']
2026-02-18T09:10:10.883 INFO:scrape:suites: ['2-ceph', '3-mgrmodules', '4-units/prometheus}}', 'debug/mgr', 'distro/{ubuntu_latest}', 'mgr_ttl_cache/disable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-hybrid}', 'tasks/{1-install']
2026-02-18T09:10:10.883 INFO:scrape: