2026-02-21T19:52:08.150 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-21T19:52:08.150 INFO:teuthology.results:subset: '1/20'
2026-02-21T19:52:08.151 INFO:teuthology.results:seed: '4054'
2026-02-21T20:08:33.749 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-21T20:08:33.749 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2026-02-22T02:01:07.760 INFO:teuthology.results:Tests finished! gathering results...
2026-02-22T02:01:10.333 INFO:scrape:Found 100 jobs
2026-02-22T02:01:13.235 INFO:scrape:Found 9 distinct failure reasons
2026-02-22T02:01:13.235 INFO:scrape:Failure: Command failed on trial051 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c \'grep \'"\'"\'Number of nodes:3 (including 1 deleted nodes)\'"\'"\' /tmp/ctdb_status\''
2026-02-22T02:01:13.235 INFO:scrape:1 jobs: ['62922']
2026-02-22T02:01:13.235 INFO:scrape:suites: ['orch:cephadm/smb/{0-distro/ubuntu_22.04', 'tasks/deploy_smb_ctdb_node_gone_state}']
2026-02-22T02:01:13.235 INFO:scrape:
2026-02-22T02:01:13.235 INFO:scrape:Failure: Command failed on trial064 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6bce4e93-0f60-11f1-906f-d404e6e7d460 -e sha1=042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''
2026-02-22T02:01:13.235 INFO:scrape:7 jobs: ['62910', '62924', '62961', '62870', '62889', '62934', '62890']
2026-02-22T02:01:13.235 INFO:scrape:suites intersection: []
2026-02-22T02:01:13.236 INFO:scrape:suites union: ['1-bootstrap/squid', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-squid', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-squid', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-tentacle', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/tentacle']
2026-02-22T02:01:13.236 INFO:scrape:
2026-02-22T02:01:13.236 INFO:scrape:Failure: Test failure: test_error (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli.test_error)
2026-02-22T02:01:13.236 INFO:scrape:2 jobs: ['62878', '62936']
2026-02-22T02:01:13.236 INFO:scrape:suites intersection: ['2-node-mgr', 'orchestrator_cli}']
2026-02-22T02:01:13.236 INFO:scrape:suites union: ['2-node-mgr', 'agent/off', 'agent/on', 'orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc}', 'orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream}', 'orchestrator_cli}']
2026-02-22T02:01:13.236 INFO:scrape:
2026-02-22T02:01:13.236 INFO:scrape:Failure: reached maximum tries (300) after waiting for 300 seconds
2026-02-22T02:01:13.236 INFO:scrape:3 jobs: ['62923', '62931', '62911']
2026-02-22T02:01:13.236 INFO:scrape:suites intersection: []
2026-02-22T02:01:13.236 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_mgmt_gateway}', 'task/test_nvmeof_basic_mtls_disabled}', 'task/test_nvmeof_mtls_enabled_cephadm_signed_certs}']
2026-02-22T02:01:13.236 INFO:scrape:
2026-02-22T02:01:13.236 INFO:scrape:Failure: Command failed on trial093 with status 1: 'sudo yum -y install ceph-radosgw'
2026-02-22T02:01:13.236 INFO:scrape:1 jobs: ['62906']
2026-02-22T02:01:13.236 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/squid']
2026-02-22T02:01:13.236 INFO:scrape:
2026-02-22T02:01:13.236 INFO:scrape:Failure: Command failed on trial080 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm'
2026-02-22T02:01:13.236 INFO:scrape:2 jobs: ['62875', '62907']
2026-02-22T02:01:13.236 INFO:scrape:suites intersection: ['1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream']
2026-02-22T02:01:13.236 INFO:scrape:suites union: ['1-bootstrap/19.2.0', '1-bootstrap/20.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream']
2026-02-22T02:01:13.236 INFO:scrape:
2026-02-22T02:01:13.236 INFO:scrape:Failure: Command failed on trial030 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd8aa3d3-0f60-11f1-938b-d404e6e7d460 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n  curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v2/status\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts | jq -e \'"\'"\'.[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n# check prometheus metrics endpoint is not empty and make sure we can get metrics\nMETRICS_URL=$(ceph mgr services | jq -r .prometheus)\n[ -n "$METRICS_URL" ] || exit 1\ncurl -s "${METRICS_URL}metrics" | grep -q \'"\'"\'^ceph_health_status\'"\'"\'\n\''
2026-02-22T02:01:13.236 INFO:scrape:1 jobs: ['62916']
2026-02-22T02:01:13.236 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'task/test_monitoring_stack_basic}']
2026-02-22T02:01:13.236 INFO:scrape:
2026-02-22T02:01:13.236 INFO:scrape:Failure: "2026-02-21T20:03:19.756478+0000 mon.a (mon.0) 324 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log
2026-02-22T02:01:13.236 INFO:scrape:1 jobs: ['62880']
2026-02-22T02:01:13.236 INFO:scrape:suites: ['cluster/{fixed-3}', 'conf/{disable-pool-app}', 'orch:cephadm/rbd_iscsi/{base/install', 'supported-container-hosts$/{ubuntu_22.04}', 'workloads/cephadm_iscsi}']
2026-02-22T02:01:13.236 INFO:scrape:
2026-02-22T02:01:13.237 INFO:scrape:Dead: b'2026-02-22T02:00:47.907 DEBUG:teuthology.exit:Finished running handlers'
2026-02-22T02:01:13.237 INFO:scrape:1 jobs: ['62900']
2026-02-22T02:01:13.237 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_iscsi_container/{centos_9.stream', 'test_iscsi_container}}']
2026-02-22T02:01:13.237 INFO:scrape:
2026-02-22T02:01:13.238 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial
=================================================================
info:    https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/
logs:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/
failed:  18
dead:    1
running: 0
waiting: 0
queued:  0
passed:  81


Fail
=================================================================
[62870]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-squid 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:18:18
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62870/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62870/

    Command failed on trial119 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell
    -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    ca28ed3c-0f5f-11f1-9e52-d404e6e7d460 -e
    sha1=042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[62875]  orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/19.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:02:35
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62875/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62875/

    Command failed on trial080 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[62878]  orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli}
-----------------------------------------------------------------
time:   00:06:08
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62878/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62878/

    Test failure: test_error
    (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli.test_error)

[62880]  orch:cephadm/rbd_iscsi/{base/install cluster/{fixed-3} conf/{disable-pool-app} supported-container-hosts$/{ubuntu_22.04} workloads/cephadm_iscsi}
-----------------------------------------------------------------
time:   00:16:47
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62880/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62880/

    "2026-02-21T20:03:19.756478+0000 mon.a (mon.0) 324 : cluster [WRN] Health
    check failed: 1 osds down (OSD_DOWN)" in cluster log

[62889]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/tentacle 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:12:36
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62889/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62889/

    Command failed on trial071 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:tentacle
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5d5f6748-0f60-11f1-beea-d404e6e7d460 -e
    sha1=042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[62890]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-tentacle 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:36:25
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62890/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62890/

    Command failed on trial063 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:tentacle
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5a8730fb-0f60-11f1-9a7f-d404e6e7d460 -e
    sha1=042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 7\'"\'"\'\''

[62906]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:02:44
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62906/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62906/

    Command failed on trial093 with status 1: 'sudo yum -y install ceph-
    radosgw'

[62907]  orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/20.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:02:27
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62907/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62907/

    Command failed on trial004 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[62910]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-squid 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:11:32
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62910/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62910/

    Command failed on trial064 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell
    -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    6bce4e93-0f60-11f1-906f-d404e6e7d460 -e
    sha1=042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[62911]  orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_mgmt_gateway}
-----------------------------------------------------------------
time:   00:19:46
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62911/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62911/

    reached maximum tries (300) after waiting for 300 seconds

[62916]  orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic}
-----------------------------------------------------------------
time:   00:13:35
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62916/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62916/

    Command failed on trial030 with status 4: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    bd8aa3d3-0f60-11f1-938b-d404e6e7d460 -- bash -c \'set -e\nset -x\nceph orch
    apply node-exporter\nceph orch apply grafana\nceph orch apply
    alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph
    orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f
    json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch
    ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r
    \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type
    prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r
    \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type
    alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r
    \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r
    --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] |
    select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host
    ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] |
    select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host
    ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] |
    select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-
    exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls
    -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n
    curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are
    responsive and database health is okay\ncurl -k -s
    https://${GRAFANA_IP}:3000/api/health\ncurl -k -s
    https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database ==
    "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch
    daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are
    responsive and mon down alert is firing\ncurl -s
    http://${PROM_IP}:9095/api/v1/status/config\ncurl -s
    http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status ==
    "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s
    http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[]
    | select(.labels | .alertname == "CephMonDown") | .state ==
    "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon
    down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v2/status\ncurl
    -s http://${ALERTM_IP}:9093/api/v2/alerts\ncurl -s
    http://${ALERTM_IP}:9093/api/v2/alerts | jq -e \'"\'"\'.[] | select(.labels
    | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n#
    check prometheus metrics endpoint is not empty and make sure we can get
    metrics\nMETRICS_URL=$(ceph mgr services | jq -r .prometheus)\n[ -n
    "$METRICS_URL" ] || exit 1\ncurl -s "${METRICS_URL}metrics" | grep -q
    \'"\'"\'^ceph_health_status\'"\'"\'\n\''

[62922]  orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_ctdb_node_gone_state}
-----------------------------------------------------------------
time:   00:12:08
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62922/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62922/

    Command failed on trial051 with status 1: 'sudo
    TESTDIR=/home/ubuntu/cephtest bash -ex -c \'grep \'"\'"\'Number of nodes:3
    (including 1 deleted nodes)\'"\'"\' /tmp/ctdb_status\''

[62923]  orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_nvmeof_basic_mtls_disabled}
-----------------------------------------------------------------
time:   00:14:37
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62923/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62923/

    reached maximum tries (300) after waiting for 300 seconds

[62924]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/tentacle 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:16:19
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62924/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62924/

    Command failed on trial016 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:tentacle
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4fe7cb01-0f60-11f1-80ab-d404e6e7d460 -e
    sha1=042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[62931]  orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_nvmeof_mtls_enabled_cephadm_signed_certs}
-----------------------------------------------------------------
time:   00:17:34
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62931/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62931/

    reached maximum tries (300) after waiting for 300 seconds

[62934]  orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/squid 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:09:52
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62934/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62934/

    Command failed on trial059 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell
    -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    39d77373-0f60-11f1-8f14-d404e6e7d460 -e
    sha1=042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[62936]  orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli}
-----------------------------------------------------------------
time:   00:07:05
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62936/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62936/

    Test failure: test_error
    (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli.test_error)

[62961]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/tentacle 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:16:34
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62961/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62961/

    Command failed on trial150 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:tentacle
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    372d57d3-0f62-11f1-9ea7-d404e6e7d460 -e
    sha1=042df16cd9ee8a8df553b188fef8a8c4a9e7ab01 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''



Dead
=================================================================
[62900]  orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62900/
log:    https://qa-proxy.ceph.com/teuthology/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62900/

    killed



Pass
=================================================================
[62871] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:07:24
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62871/

[62872] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:07:13
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62872/

[62873] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos}
time:   00:03:27
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62873/

[62874] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_ctdb_res_ips}
time:   00:08:15
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62874/

[62876] orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} nfs-version/{4.1} overrides/{ignore_mgr_down ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs}
time:   00:30:21
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62876/

[62877] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli}
time:   00:06:49
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62877/

[62879] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:06:15
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62879/

[62881] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:07:09
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62881/

[62882] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final}
time:   00:04:56
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62882/

[62883] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:05:33
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62883/

[62884] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_ctdb_res_ports2c}
time:   00:09:52
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62884/

[62885] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:07:24
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62885/

[62886] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_extra_daemon_features}
time:   00:07:07
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62886/

[62887] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_domain}
time:   00:15:02
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62887/

[62888] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
time:   00:06:24
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62888/

[62891] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:05:37
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62891/

[62892] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_res_basic}
time:   00:12:13
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62892/

[62893] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
time:   00:20:18
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62893/

[62894] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
time:   00:17:06
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62894/

[62895] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_host_drain}
time:   00:08:15
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62895/

[62896] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:06:50
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62896/

[62897] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_res_dom}
time:   00:12:31
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62897/

[62898] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon}
time:   00:13:27
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62898/

[62899] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:06:03
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62899/

[62901] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:19:13
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62901/

[62902] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_ports}
time:   00:08:25
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62902/

[62903] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final}
time:   00:05:57
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62903/

[62904] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:06:32
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62904/

[62905] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_proxy_disabled}
time:   00:06:40
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62905/

[62908] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:07:39
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62908/

[62909] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:05:49
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62909/

[62912] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_proxy_enabled}
time:   00:06:27
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62912/

[62913] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:07:23
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62913/

[62914] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root}
time:   00:27:51
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62914/

[62915] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python}
time:   00:14:49
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62915/

[62917] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption}
time:   00:04:12
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62917/

[62918] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw}
time:   00:06:00
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62918/

[62919] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic}
time:   00:07:00
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62919/

[62920] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:05:36
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62920/

[62921] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:05:55
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62921/

[62925] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-tentacle 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
time:   00:32:24
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62925/

[62926] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:06:06
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62926/

[62927] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain}
time:   00:05:04
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62927/

[62928] orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:07:11
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62928/

[62929] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:05:56
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62929/

[62930] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:06:12
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62930/

[62932] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_basic}
time:   00:12:17
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62932/

[62933] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys}
time:   00:25:44
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62933/

[62935] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout}
time:   00:08:41
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62935/

[62937] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:06:01
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62937/

[62938] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final}
time:   00:05:36
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62938/

[62939] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:06:10
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62939/

[62940] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_clustering_ips}
time:   00:12:52
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62940/

[62941] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite}
time:   00:07:44
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62941/

[62942] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
time:   00:06:21
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62942/

[62943] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:05:51
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62943/

[62944] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_ctdb_res_basic}
time:   00:08:07
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62944/

[62945] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:14:08
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62945/

[62946] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-squid 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic}
time:   00:25:15
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62946/

[62947] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:07:01
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62947/

[62948] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations}
time:   00:09:46
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62948/

[62949] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_ctdb_res_dom}
time:   00:08:29
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62949/

[62950] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:06:37
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62950/

[62951] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_ctdb_res_ips}
time:   00:12:29
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62951/

[62952] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli}
time:   00:07:15
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62952/

[62953] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:06:14
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62953/

[62954] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:07:20
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62954/

[62955] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:07:20
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62955/

[62956] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
time:   00:30:46
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62956/

[62957] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests}
time:   00:19:42
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62957/

[62958] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key}
time:   00:05:56
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62958/

[62959] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_ctdb_res_ports2c}
time:   00:08:58
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62959/

[62960] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:07:43
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62960/

[62962] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-tentacle 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
time:   00:34:16
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62962/

[62963] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_domain}
time:   00:12:40
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62963/

[62964] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm}
time:   00:06:52
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62964/

[62965] orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/tentacle 1-start 2-nfs 3-upgrade-with-workload 4-final}
time:   00:17:37
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62965/

[62966] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
time:   00:06:18
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62966/

[62967] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:05:43
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62967/

[62968] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:05:54
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62968/

[62969] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_basic}
time:   00:14:35
info:   https://pulpito.ceph.com/adking-2026-02-21_19:52:04-orch:cephadm-wip-adk3-testing-2026-02-12-2007-distro-default-trial/62969/
2026-02-22T02:01:13.245 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 33, in main
    results(args['--archive-dir'], args['--name'], args['--email'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 78, in results
    email_results(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 95, in email_results
    smtp = smtplib.SMTP('localhost')
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 341, in connect
    self.sock = self._get_socket(host, port, self.timeout)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket
    return socket.create_connection((host, port), timeout,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection
    sock.connect(sa)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect
    self._internal_connect(address)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect
    raise _SocketError(err, strerror(err))
ConnectionRefusedError: [Errno 111] Connection refused