2026-02-25T11:41:14.588 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-25T11:41:14.588 INFO:teuthology.results:subset: '111/120000'
2026-02-25T11:41:14.588 INFO:teuthology.results:seed: '6208'
2026-02-25T11:51:09.515 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-25T11:51:09.521 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2026-02-25T12:16:11.095 INFO:teuthology.results:Tests finished! gathering results...
2026-02-25T12:16:11.496 INFO:scrape:Found 26 jobs
2026-02-25T12:16:11.848 INFO:scrape:Found 6 distinct failure reasons
2026-02-25T12:16:11.849 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on trial016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=506b402a3d3a240d0bfe073b4769eaffdf84497b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'
2026-02-25T12:16:11.849 INFO:scrape:2 jobs: ['69865', '69841']
2026-02-25T12:16:11.849 INFO:scrape:suites intersection: ['agent/on', 'task/test_cephadm_repos}']
2026-02-25T12:16:11.849 INFO:scrape:suites union: ['agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados:cephadm/workunits/{0-distro/rocky_10', 'rados:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}']
2026-02-25T12:16:11.849 INFO:scrape:
2026-02-25T12:16:11.849 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on trial015 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=506b402a3d3a240d0bfe073b4769eaffdf84497b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2026-02-25T12:16:11.849 INFO:scrape:1 jobs: ['69863']
2026-02-25T12:16:11.849 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados:cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_cephadm}']
2026-02-25T12:16:11.849 INFO:scrape:
2026-02-25T12:16:11.849 INFO:scrape:Failure: "2026-02-25T12:00:21.592407+0000 mon.a (mon.0) 446 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log
2026-02-25T12:16:11.849 INFO:scrape:1 jobs: ['69855']
2026-02-25T12:16:11.849 INFO:scrape:suites: ['agent/on', 'mon_election/classic', 'rados:cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_rgw_multisite}']
2026-02-25T12:16:11.849 INFO:scrape:
2026-02-25T12:16:11.849 INFO:scrape:Failure: Command failed (workunit test cephadm/test_iscsi_setup.sh) on trial055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=506b402a3d3a240d0bfe073b4769eaffdf84497b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_iscsi_setup.sh'
2026-02-25T12:16:11.849 INFO:scrape:1 jobs: ['69849']
2026-02-25T12:16:11.849 INFO:scrape:suites: ['agent/off', 'mon_election/connectivity', 'rados:cephadm/workunits/{0-distro/rocky_10', 'task/test_iscsi_container/{centos_9.stream', 'test_iscsi_container}}']
2026-02-25T12:16:11.849 INFO:scrape:
2026-02-25T12:16:11.849 INFO:scrape:Failure: Command failed on trial143 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:506b402a3d3a240d0bfe073b4769eaffdf84497b-rockylinux-10 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b72a921-1241-11f1-b25b-d404e6e7d460 -- bash -c \'set -ex\n# Deploy monitoring stack\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\n# generate SSL certificate\nopenssl req -x509 -newkey rsa:4096 -keyout /tmp/key.pem -out /tmp/cert.pem -sha256 -days 30 -nodes -subj "/CN=*"\n# Generate a mgmt.spec template\ncat << EOT > /tmp/mgmt.spec\nservice_type: mgmt-gateway\nservice_id: foo\nplacement:\n  hosts:\n    - ${HOSTNAME}\nspec:\n  ssl_protocols:\n    - TLSv1.2\n    - TLSv1.3\n  ssl_ciphers:\n    - AES128-SHA\n    - AES256-SHA\n  enable_health_check_endpoint: True\nEOT\n# Add generated certificates to spec file\necho "  ssl_cert: |" >> /tmp/mgmt.spec\nwhile read LINE; do echo $LINE | sed -e "s/^/    /"; done < /tmp/cert.pem >> /tmp/mgmt.spec\necho "  ssl_key: |" >> /tmp/mgmt.spec\nwhile read LINE; do echo $LINE | sed -e "s/^/    /"; done < /tmp/key.pem >> /tmp/mgmt.spec\n# Apply spec\nceph orch apply -i /tmp/mgmt.spec\n\''
2026-02-25T12:16:11.849 INFO:scrape:1 jobs: ['69851']
2026-02-25T12:16:11.849 INFO:scrape:suites: ['agent/on', 'mon_election/classic', 'rados:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_mgmt_gateway}']
2026-02-25T12:16:11.849 INFO:scrape:
2026-02-25T12:16:11.849 INFO:scrape:Failure: Command failed on trial089 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:506b402a3d3a240d0bfe073b4769eaffdf84497b-rockylinux-10 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b0080bee-1241-11f1-a616-d404e6e7d460 -- bash -c \'set -ex\n# since we don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in order to replace the fake hostnames "host.a/b/c" with\n# the actual names cephadm knows the host by within the mon spec\nceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'a\\nb\\nc\'"\'"\' > mon_ids\necho $\'"\'"\'{datacenter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' > crush_locs\nceph orch ls --service-name mon --export > mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3 && read fakename <&4; do\n  MONSPEC="${MONSPEC//$fakename/$realname}"\ndone 3<realnames 4<fakenames\necho "$MONSPEC" > mon.yaml\ncat mon.yaml\n# now the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep 90\nceph mon dump\nceph mon dump --format json\n# verify all the crush locations got set from "ceph mon dump" output\nwhile read monid <&3 && read crushloc <&4; do\n  ceph mon dump --format json | jq --arg monid "$monid" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] | select(.name == $monid) | .crush_location == $crushloc\'"\'"\'\ndone 3<mon_ids 4<crush_locs\n\''
2026-02-25T12:16:11.849 INFO:scrape:1 jobs: ['69858']
2026-02-25T12:16:11.849 INFO:scrape:suites: ['agent/off', 'mon_election/connectivity', 'rados:cephadm/workunits/{0-distro/rocky_10', 'task/test_set_mon_crush_locations}']
2026-02-25T12:16:11.850 INFO:scrape: