2024-04-02T04:11:46.387 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-02T04:11:46.387 INFO:teuthology.results:subset: '52/64'
2024-04-02T04:11:46.388 INFO:teuthology.results:seed: '6530'
2024-04-02T06:34:43.289 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-02T06:34:43.309 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-02T18:34:46.551 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2024-04-02T18:34:59.331 INFO:scrape:Found 171 jobs
2024-04-02T18:35:03.851 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.852 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.852 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.853 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.853 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.853 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.853 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.853 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.854 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.854 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.854 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.854 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:03.854 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:09.596 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:09.597 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:09.597 WARNING:scrape:Saw end of BT but not start
2024-04-02T18:35:31.478 INFO:scrape:Found 10 distinct failure reasons
2024-04-02T18:35:31.478 INFO:scrape:Failure: "2024-04-02T04:35:00.877696+0000 mon.smithi106 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log
2024-04-02T18:35:31.478 INFO:scrape:90 jobs
2024-04-02T18:35:31.479 INFO:scrape:suites intersection: []
2024-04-02T18:35:31.479 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-start', '2-nfs', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/mirror', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'conf/{disable-pool-app}', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch/cephadm/osds/{0-distro/centos_9.stream', 'orch/cephadm/osds/{0-distro/centos_9.stream_runc', 'orch/cephadm/osds/{0-distro/ubuntu_22.04', 'orch/cephadm/rbd_iscsi/{0-single-container-host', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch/cephadm/thrash/{0-distro/centos_9.stream', 'orch/cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch/cephadm/thrash/{0-distro/ubuntu_22.04', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch/cephadm/with-work/{0-distro/centos_9.stream', 'orch/cephadm/with-work/{0-distro/centos_9.stream_runc', 'orch/cephadm/with-work/{0-distro/ubuntu_22.04', 'root}', 'start', 'start}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'workloads/cephadm_iscsi}']
2024-04-02T18:35:31.479 INFO:scrape:
2024-04-02T18:35:31.479 INFO:scrape:Failure: Command failed on smithi189 with status 1: 'sudo yum install -y kernel'
2024-04-02T18:35:31.480 INFO:scrape:1 jobs: ['7634955']
2024-04-02T18:35:31.480 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-04-02T18:35:31.480 INFO:scrape:
2024-04-02T18:35:31.480 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-04-02T18:35:31.480 INFO:scrape:6 jobs: ['7634908', '7634838', '7634882', '7634936', '7634848', '7634982']
2024-04-02T18:35:31.480 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}']
2024-04-02T18:35:31.480 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.1}']
2024-04-02T18:35:31.480 INFO:scrape:
2024-04-02T18:35:31.481 INFO:scrape:Dead: b'2024-04-02T17:08:43.246 DEBUG:teuthology.exit:Finished running handlers'
2024-04-02T18:35:31.481 INFO:scrape:4 jobs: ['7634899', '7634902', '7634868', '7634878']
2024-04-02T18:35:31.481 INFO:scrape:suites intersection: []
2024-04-02T18:35:31.481 INFO:scrape:suites union: ['0-nvme-loop', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '2-repo_digest/defaut', '2-services/nvmeof', '3-final}', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade/simple', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fixed-2', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon_election/classic', 'mon_election/connectivity}', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'start}', 'syntax}', 'tasks/{0-from/quincy']
2024-04-02T18:35:31.481 INFO:scrape:
2024-04-02T18:35:31.481 INFO:scrape:Failure: Command failed on smithi105 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"
2024-04-02T18:35:31.481 INFO:scrape:4 jobs: ['7634993', '7634909', '7634951', '7634867']
2024-04-02T18:35:31.481 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_22.04']
2024-04-02T18:35:31.481 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_22.04', 'rook/1.7.2}', 'rook/master}']
2024-04-02T18:35:31.481 INFO:scrape:
2024-04-02T18:35:31.482 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6cec793d9073f0594e2364b9b687ecaf22424cca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2024-04-02T18:35:31.482 INFO:scrape:3 jobs: ['7634829', '7634891', '7634961']
2024-04-02T18:35:31.482 INFO:scrape:suites intersection: ['task/test_cephadm}']
2024-04-02T18:35:31.482 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_9.stream', 'orch/cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}']
2024-04-02T18:35:31.482 INFO:scrape:
2024-04-02T18:35:31.482 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
2024-04-02T18:35:31.482 INFO:scrape:2 jobs: ['7634894', '7634923']
2024-04-02T18:35:31.482 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-04-02T18:35:31.482 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-04-02T18:35:31.482 INFO:scrape:
2024-04-02T18:35:31.483 INFO:scrape:Failure: Failed to reconnect to smithi189
2024-04-02T18:35:31.483 INFO:scrape:1 jobs: ['7634956']
2024-04-02T18:35:31.483 INFO:scrape:suites: ['mon_election/connectivity', 'orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc', 'task/test_orch_cli_mon}']
2024-04-02T18:35:31.483 INFO:scrape:
2024-04-02T18:35:31.483 INFO:scrape:Failure: Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a781000-f0b0-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''
2024-04-02T18:35:31.483 INFO:scrape:2 jobs: ['7634904', '7634976']
2024-04-02T18:35:31.483 INFO:scrape:suites intersection: ['2-repo_digest/repo_digest', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream']
2024-04-02T18:35:31.483 INFO:scrape:suites union: ['2-repo_digest/repo_digest', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream']
2024-04-02T18:35:31.483 INFO:scrape:
2024-04-02T18:35:31.483 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6cec793d9073f0594e2364b9b687ecaf22424cca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'
2024-04-02T18:35:31.483 INFO:scrape:1 jobs: ['7634967']
2024-04-02T18:35:31.484 INFO:scrape:suites: ['agent/on', 'mon_election/classic', 'orch/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}']
2024-04-02T18:35:31.484 INFO:scrape:
2024-04-02T18:35:31.489 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/
failed:  110
dead:    3
running: 1
waiting: 0
queued:  0
passed:  57


Fail
=================================================================
[7634828]  orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   01:01:50
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634828/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634828/

    "2024-04-02T04:34:39.259094+0000 mon.a (mon.0) 199 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7634829]  orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:09:23
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634829/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634829/
sentry: https://sentry.ceph.com/organizations/ceph/?query=748fcd082f544a0e8a7c72f7691b44bb

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi052 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6cec793d9073f0594e2364b9b687ecaf22424cca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[7634830]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
-----------------------------------------------------------------
time:   00:23:30
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634830/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634830/

    "2024-04-02T04:35:00.877696+0000 mon.smithi106 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634831]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:56:30
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634831/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634831/

    "2024-04-02T04:35:37.362303+0000 mon.a (mon.0) 478 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7634832]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:15:15
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634832/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634832/

    "2024-04-02T04:36:44.312226+0000 mon.smithi049 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634836]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final}
-----------------------------------------------------------------
time:   00:14:58
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634836/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634836/

    "2024-04-02T04:33:58.349695+0000 mon.smithi170 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634837]  orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:47:42
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634837/

    "2024-04-02T04:36:04.308495+0000 mon.a (mon.0) 195 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7634838]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:37:46
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634838/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634838/

    reached maximum tries (51) after waiting for 300 seconds

[7634840]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final}
-----------------------------------------------------------------
time:   00:20:01
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634840/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634840/

    "2024-04-02T04:37:17.063430+0000 mon.smithi169 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634841]  orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
-----------------------------------------------------------------
time:   00:21:05
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634841/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634841/

    "2024-04-02T04:37:26.024468+0000 mon.smithi103 (mon.0) 261 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634843]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final}
-----------------------------------------------------------------
time:   00:14:42
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634843/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634843/

    "2024-04-02T04:34:54.321517+0000 mon.smithi017 (mon.0) 273 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634845]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
-----------------------------------------------------------------
time:   00:18:36
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634845/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634845/

    "2024-04-02T04:35:59.316296+0000 mon.smithi071 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634847]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final}
-----------------------------------------------------------------
time:   00:21:50
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634847/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634847/

    "2024-04-02T04:42:37.569690+0000 mon.smithi037 (mon.0) 261 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634848]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:38:49
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634848/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634848/

    reached maximum tries (51) after waiting for 300 seconds

[7634849]  orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:33:54
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849/

    "2024-04-02T04:46:49.936349+0000 mon.smithi053 (mon.0) 872 : cluster [WRN]
    Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in
    cluster log

[7634853]  orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all}
-----------------------------------------------------------------
time:   00:12:53
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634853/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634853/

    "2024-04-02T04:40:49.096638+0000 mon.smithi144 (mon.0) 268 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634854]  orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi}
-----------------------------------------------------------------
time:   00:25:06
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634854/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634854/

    "2024-04-02T04:45:36.997362+0000 mon.a (mon.0) 209 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7634856]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final}
-----------------------------------------------------------------
time:   00:13:28
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634856/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634856/

    "2024-04-02T04:46:13.663352+0000 mon.smithi045 (mon.0) 271 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634858]  orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:11:42
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634858/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634858/

    "2024-04-02T04:47:39.252164+0000 mon.a (mon.0) 210 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7634859]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:46:48
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634859/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634859/

    "2024-04-02T04:58:27.647337+0000 mon.a (mon.0) 946 : cluster [WRN] Health
    check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in
    cluster log

[7634860]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final}
-----------------------------------------------------------------
time:   00:13:55
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634860/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634860/

    "2024-04-02T04:47:17.375525+0000 mon.smithi131 (mon.0) 269 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634862]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final}
-----------------------------------------------------------------
time:   00:20:20
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634862/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634862/

    "2024-04-02T04:51:52.512462+0000 mon.smithi079 (mon.0) 263 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634864]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final}
-----------------------------------------------------------------
time:   00:14:28
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634864/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634864/

    "2024-04-02T04:49:06.009490+0000 mon.smithi150 (mon.0) 267 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634866]  orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:14:06
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634866/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634866/

    "2024-04-02T04:50:50.604412+0000 mon.smithi023 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634867]  orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/host rook/master}
-----------------------------------------------------------------
time:   00:07:14
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634867/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634867/
sentry: https://sentry.ceph.com/organizations/ceph/?query=abc48b8a32b949b78b617305eeedcf55

    Command failed on smithi102 with status 100: "sudo apt update && sudo apt
    install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo
    /usr/share/keyrings/kubernetes-archive-keyring.gpg
    https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-
    by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]
    https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee
    /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt
    install -y kubelet kubeadm kubectl bridge-utils"

[7634870]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final}
-----------------------------------------------------------------
time:   00:14:39
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634870/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634870/

    "2024-04-02T04:50:30.589995+0000 mon.smithi059 (mon.0) 271 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634871]  orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:32:05
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634871/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634871/

    "2024-04-02T05:00:00.000146+0000 mon.a (mon.0) 1051 : cluster 3 [WRN]
    OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

[7634872]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
-----------------------------------------------------------------
time:   00:21:25
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634872/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634872/

    "2024-04-02T05:00:00.000264+0000 mon.smithi003 (mon.0) 790 : cluster 4
    [ERR] MDS_ALL_DOWN: 1 filesystem is offline" in cluster log

[7634875]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
-----------------------------------------------------------------
time:   00:15:27
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634875/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634875/

    "2024-04-02T04:52:00.399168+0000 mon.smithi100 (mon.0) 269 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634876]  orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:12:13
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634876/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634876/

    "2024-04-02T04:52:29.496724+0000 mon.a (mon.0) 267 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7634877]  orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag}
-----------------------------------------------------------------
time:   00:19:56
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634877/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634877/

    "2024-04-02T05:00:10.471865+0000 mon.smithi026 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634879]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
-----------------------------------------------------------------
time:   00:15:55
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634879/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634879/

    "2024-04-02T05:02:37.989488+0000 mon.smithi164 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634880]  orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   01:01:01
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634880/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634880/

    "2024-04-02T05:07:44.265958+0000 mon.a (mon.0) 198 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7634882]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:41:46
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634882/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634882/

    reached maximum tries (51) after waiting for 300 seconds

[7634884]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
-----------------------------------------------------------------
time:   00:23:35
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634884/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634884/

    "2024-04-02T05:06:26.935906+0000 mon.smithi170 (mon.0) 264 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634885]  orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:49:32
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634885/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634885/

    "2024-04-02T05:07:53.666500+0000 mon.a (mon.0) 198 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7634886]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
-----------------------------------------------------------------
time:   00:16:13
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634886/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634886/

    "2024-04-02T05:02:05.060118+0000 mon.smithi138 (mon.0) 269 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634888]  orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait}
-----------------------------------------------------------------
time:   00:12:58
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634888/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634888/

    "2024-04-02T05:02:19.527575+0000 mon.smithi031 (mon.0) 268 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634889]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:14:58
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634889/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634889/

    "2024-04-02T05:05:33.881163+0000 mon.smithi033 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634890]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final}
-----------------------------------------------------------------
time:   00:21:15
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634890/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634890/

    "2024-04-02T05:10:50.014102+0000 mon.smithi012 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634891]  orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_cephadm}
-----------------------------------------------------------------
time:   00:11:49
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634891/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634891/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a282c03fde084557ab2ff5bd300eb995

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi062 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6cec793d9073f0594e2364b9b687ecaf22424cca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[7634893]  orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python}
-----------------------------------------------------------------
time:   00:22:45
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634893/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634893/

    "2024-04-02T05:20:00.000234+0000 mon.a (mon.0) 1081 : cluster 3 [WRN]
    POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in
    cluster log

[7634894]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   03:40:26
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634894/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634894/
sentry: https://sentry.ceph.com/organizations/ceph/?query=75c8a5893314457cbb43a0fdd5a40a1a

    Command failed (workunit test suites/fsstress.sh) on smithi123 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6cec793d9073f0594e2364b9b687ecaf22424cca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[7634897]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final}
-----------------------------------------------------------------
time:   00:14:19
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634897/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634897/

    "2024-04-02T05:19:59.717998+0000 mon.smithi019 (mon.0) 273 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634900]  orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
-----------------------------------------------------------------
time:   00:14:29
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634900/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634900/

    "2024-04-02T05:19:22.297539+0000 mon.smithi153 (mon.0) 269 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634901]  orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:49:58
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634901/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634901/

    "2024-04-02T05:23:42.844120+0000 mon.a (mon.0) 203 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7634904]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:42:25
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634904/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634904/
sentry: https://sentry.ceph.com/organizations/ceph/?query=07f912b12f9443cabd5d584916f4bf2d

    Command failed on smithi037 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4a781000-f0b0-11ee-b647-cb9ed24678a4 -e
    sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 7\'"\'"\'\''

[7634905]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
-----------------------------------------------------------------
time:   00:26:12
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634905/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634905/

    "2024-04-02T05:24:46.343542+0000 mon.smithi023 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634906]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final}
-----------------------------------------------------------------
time:   00:15:25
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634906/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634906/

    "2024-04-02T05:18:07.532693+0000 mon.smithi059 (mon.0) 267 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634908]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:54:41
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634908/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634908/

    reached maximum tries (51) after waiting for 300 seconds

[7634909]  orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/calico rook/1.7.2}
-----------------------------------------------------------------
time:   00:07:25
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634909/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634909/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bafd2491d3be4adfba13bfd5875dc79a

    Command failed on smithi181 with status 100: "sudo apt update && sudo apt
    install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo
    /usr/share/keyrings/kubernetes-archive-keyring.gpg
    https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-
    by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]
    https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee
    /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt
    install -y kubelet kubeadm kubectl bridge-utils"

[7634911]  orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all}
-----------------------------------------------------------------
time:   00:18:45
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634911/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634911/

    "2024-04-02T05:23:33.258564+0000 mon.smithi131 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634912]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final}
-----------------------------------------------------------------
time:   00:12:54
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634912/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634912/

    "2024-04-02T05:20:29.488022+0000 mon.smithi148 (mon.0) 269 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634914]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final}
-----------------------------------------------------------------
time:   00:19:59
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634914/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634914/

    "2024-04-02T05:25:28.131722+0000 mon.smithi089 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634915]  orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   01:11:57
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634915/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634915/

    "2024-04-02T05:40:00.000175+0000 mon.a (mon.0) 1378 : cluster 3 [WRN]
    OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

[7634918]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final}
-----------------------------------------------------------------
time:   00:14:09
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634918/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634918/

    "2024-04-02T05:28:59.668079+0000 mon.smithi169 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634919]  orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:11:20
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634919/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634919/

    "2024-04-02T05:31:12.383784+0000 mon.a (mon.0) 210 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7634920]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final}
-----------------------------------------------------------------
time:   00:14:25
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634920/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634920/

    "2024-04-02T05:30:07.782029+0000 mon.smithi064 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634921]  orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:13:50
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634921/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634921/

    "2024-04-02T05:36:40.911921+0000 mon.smithi098 (mon.0) 269 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634923]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   03:36:56
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634923/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634923/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b3b8be91c9774ee2b2ea68e4e9f6b0bd

    Command failed (workunit test suites/fsstress.sh) on smithi203 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6cec793d9073f0594e2364b9b687ecaf22424cca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[7634925]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final}
-----------------------------------------------------------------
time:   00:20:35
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634925/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634925/

    "2024-04-02T05:38:01.068377+0000 mon.smithi053 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634926]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:37:48
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634926/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634926/

    "2024-04-02T05:35:34.930035+0000 mon.a (mon.0) 169 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7634927]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
-----------------------------------------------------------------
time:   00:14:31
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634927/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634927/

    "2024-04-02T05:33:48.802082+0000 mon.smithi016 (mon.0) 273 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634928]  orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:30:26
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634928/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634928/

    "2024-04-02T05:50:00.000120+0000 mon.a (mon.0) 1085 : cluster 3 [WRN]
    OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

[7634929]  orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:51:30
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634929/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634929/

    "2024-04-02T05:50:00.000228+0000 mon.a (mon.0) 2567 : cluster 3 [WRN]
    POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in
    cluster log

[7634931]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
-----------------------------------------------------------------
time:   00:15:08
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634931/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634931/

    "2024-04-02T05:35:19.902761+0000 mon.smithi177 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634932]  orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag}
-----------------------------------------------------------------
time:   00:13:07
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634932/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634932/

    "2024-04-02T05:34:07.558070+0000 mon.smithi083 (mon.0) 272 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634933]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
-----------------------------------------------------------------
time:   00:22:04
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634933/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634933/

    "2024-04-02T05:37:23.565915+0000 mon.smithi082 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634936]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:07:38
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634936/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634936/

    reached maximum tries (51) after waiting for 300 seconds

[7634937]  orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:35:33
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634937/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634937/

    "2024-04-02T05:48:04.661925+0000 mon.smithi072 (mon.0) 869 : cluster [WRN]
    Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in
    cluster log

[7634942]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
-----------------------------------------------------------------
time:   00:16:50
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634942/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634942/

    "2024-04-02T05:47:31.038583+0000 mon.smithi106 (mon.0) 272 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634944]  orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:11:25
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634944/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634944/

    "2024-04-02T05:49:03.429231+0000 mon.a (mon.0) 212 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7634945]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
-----------------------------------------------------------------
time:   00:17:20
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634945/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634945/

    "2024-04-02T05:51:04.012322+0000 mon.smithi066 (mon.0) 271 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634947]  orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait}
-----------------------------------------------------------------
time:   00:19:19
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634947/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634947/

    "2024-04-02T05:53:35.947427+0000 mon.smithi059 (mon.0) 264 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634948]  orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:51:28
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634948/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634948/

    "2024-04-02T05:54:26.126290+0000 mon.a (mon.0) 201 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7634949]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:21:20
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634949/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634949/

    "2024-04-02T05:53:16.751277+0000 mon.smithi032 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634950]  orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:32:24
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634950/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634950/

    "2024-04-02T05:54:12.121538+0000 mon.a (mon.0) 201 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7634951]  orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/flannel rook/master}
-----------------------------------------------------------------
time:   00:07:54
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634951/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634951/
sentry: https://sentry.ceph.com/organizations/ceph/?query=99c00b4826e8409dadb0aba217a85674

    Command failed on smithi029 with status 100: "sudo apt update && sudo apt
    install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo
    /usr/share/keyrings/kubernetes-archive-keyring.gpg
    https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-
    by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]
    https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee
    /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt
    install -y kubelet kubeadm kubectl bridge-utils"

[7634952]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final}
-----------------------------------------------------------------
time:   00:15:06
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634952/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634952/

    "2024-04-02T05:49:30.916646+0000 mon.smithi132 (mon.0) 269 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634953]  orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:37:39
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634953/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634953/

    "2024-04-02T05:51:47.465545+0000 mon.a (mon.0) 169 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7634955]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634955/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634955/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e4ddc6ed6814330ab3ad012b61f80bf

    Command failed on smithi189 with status 1: 'sudo yum install -y kernel'

[7634956]  orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634956/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634956/
sentry: https://sentry.ceph.com/organizations/ceph/?query=207582f2a85543daa14a67bc3c63c9e6

    Failed to reconnect to smithi189

[7634957]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final}
-----------------------------------------------------------------
time:   00:14:07
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634957/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634957/

    "2024-04-02T05:54:26.120066+0000 mon.smithi084 (mon.0) 273 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634958]  orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
-----------------------------------------------------------------
time:   00:14:04
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634958/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634958/

    "2024-04-02T06:01:27.866661+0000 mon.smithi050 (mon.0) 266 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634959]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final}
-----------------------------------------------------------------
time:   00:19:43
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634959/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634959/

    "2024-04-02T06:24:46.926291+0000 mon.smithi006 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634961]  orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:10:12
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634961/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634961/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ef67332e1551482daa1fe63e77ddd742

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi046 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6cec793d9073f0594e2364b9b687ecaf22424cca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[7634962]  orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:37:19
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634962/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634962/

    "2024-04-02T06:40:00.070124+0000 mon.a (mon.0) 2057 : cluster 3 [WRN]
    OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

[7634963]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
-----------------------------------------------------------------
time:   00:17:54
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634963/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634963/

    "2024-04-02T06:28:05.763457+0000 mon.smithi003 (mon.0) 271 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634964]  orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:12:44
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634964/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634964/

    "2024-04-02T06:35:02.498004+0000 mon.a (mon.0) 267 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7634966]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final}
-----------------------------------------------------------------
time:   00:16:10
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634966/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634966/

    "2024-04-02T06:35:25.489184+0000 mon.smithi062 (mon.0) 274 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634967]  orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_cephadm_repos}
-----------------------------------------------------------------
time:   00:09:13
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634967/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634967/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3d5a819cebaf4f6f9b33c1cdfd8033e6

    Command failed (workunit test cephadm/test_repos.sh) on smithi027 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6cec793d9073f0594e2364b9b687ecaf22424cca
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

[7634970]  orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all}
-----------------------------------------------------------------
time:   00:12:48
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634970/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634970/

    "2024-04-02T06:37:34.361889+0000 mon.smithi012 (mon.0) 268 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634971]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final}
-----------------------------------------------------------------
time:   00:19:15
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634971/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634971/

    "2024-04-02T06:39:00.286175+0000 mon.smithi112 (mon.0) 261 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634972]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final}
-----------------------------------------------------------------
time:   00:14:33
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634972/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634972/

    "2024-04-02T06:36:35.540563+0000 mon.smithi070 (mon.0) 270 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634974]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final}
-----------------------------------------------------------------
time:   00:14:46
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634974/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634974/

    "2024-04-02T06:37:19.369155+0000 mon.smithi033 (mon.0) 272 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634975]  orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:54:06
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634975/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634975/

    "2024-04-02T07:00:00.000142+0000 mon.a (mon.0) 1476 : cluster 3 [WRN]
    OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

[7634976]  orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:43:43
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634976/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634976/
sentry: https://sentry.ceph.com/organizations/ceph/?query=81d3c7ab1591432db02bca18bc6819f3

    Command failed on smithi016 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    fcace7fa-f0ba-11ee-b647-cb9ed24678a4 -e
    sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 7\'"\'"\'\''

[7634978]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final}
-----------------------------------------------------------------
time:   00:23:19
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634978/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634978/

    "2024-04-02T06:41:43.977293+0000 mon.smithi176 (mon.0) 262 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634979]  orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:19:23
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634979/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634979/

    "2024-04-02T06:39:04.212840+0000 mon.smithi082 (mon.0) 263 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634980]  orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:51:34
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634980/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634980/

    "2024-04-02T06:50:00.000167+0000 mon.a (mon.0) 2411 : cluster 3 [WRN]
    POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in
    cluster log

[7634982]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:39:22
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634982/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634982/

    reached maximum tries (51) after waiting for 300 seconds

[7634985]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final}
-----------------------------------------------------------------
time:   00:14:28
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634985/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634985/

    "2024-04-02T06:36:54.648458+0000 mon.smithi129 (mon.0) 269 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634987]  orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:12:27
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634987/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634987/

    "2024-04-02T06:39:00.454956+0000 mon.a (mon.0) 268 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7634988]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
-----------------------------------------------------------------
time:   00:15:10
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634988/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634988/

    "2024-04-02T06:42:24.944060+0000 mon.smithi042 (mon.0) 272 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634990]  orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
-----------------------------------------------------------------
time:   00:21:58
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634990/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634990/

    "2024-04-02T06:51:00.660072+0000 mon.smithi037 (mon.0) 271 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634991]  orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag}
-----------------------------------------------------------------
time:   00:13:47
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634991/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634991/

    "2024-04-02T06:48:41.978186+0000 mon.smithi098 (mon.0) 273 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634992]  orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:42:04
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634992/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634992/

    "2024-04-02T06:53:27.506017+0000 mon.a (mon.0) 201 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7634993]  orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/host rook/1.7.2}
-----------------------------------------------------------------
time:   00:07:20
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634993/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634993/
sentry: https://sentry.ceph.com/organizations/ceph/?query=714bef802d014849a81ea560f39cbf95

    Command failed on smithi105 with status 100: "sudo apt update && sudo apt
    install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo
    /usr/share/keyrings/kubernetes-archive-keyring.gpg
    https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-
    by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]
    https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee
    /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt
    install -y kubelet kubeadm kubectl bridge-utils"

[7634994]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
-----------------------------------------------------------------
time:   00:15:17
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634994/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634994/

    "2024-04-02T06:51:38.991261+0000 mon.smithi143 (mon.0) 271 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log

[7634998]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
-----------------------------------------------------------------
time:   00:17:26
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634998/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634998/

    "2024-04-02T06:51:40.018381+0000 mon.smithi046 (mon.0) 272 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster
    log



Dead
=================================================================
[7634868]  orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634868/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634868/

    hit max job timeout

[7634899]  orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634899/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634899/

    hit max job timeout

[7634902]  orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634902/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634902/

    hit max job timeout



Running
=================================================================
[7634878] orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634878/



Pass
=================================================================
[7634833] orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:11:14
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634833/

[7634834] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:25:26
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634834/

[7634835] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos}
time:   00:07:14
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634835/

[7634839] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_orch_cli_mon}
time:   00:38:00
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634839/

[7634842] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_extra_daemon_features}
time:   00:19:31
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634842/

[7634844] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_python}
time:   00:22:01
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634844/

[7634846] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_host_drain}
time:   00:17:38
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634846/

[7634850] orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs}
time:   00:46:05
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634850/

[7634851] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_adoption}
time:   00:10:45
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634851/

[7634852] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli}
time:   00:12:13
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634852/

[7634855] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:14:38
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634855/

[7634857] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final}
time:   00:11:44
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634857/

[7634861] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}}
time:   00:18:02
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634861/

[7634863] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rotate-keys}
time:   00:23:33
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634863/

[7634865] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_monitoring_stack_basic}
time:   00:25:24
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634865/

[7634869] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_cephadm_timeout}
time:   00:13:44
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634869/

[7634873] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_rgw_multisite}
time:   00:17:43
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634873/

[7634874] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:15:21
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634874/

[7634881] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_set_mon_crush_locations}
time:   00:15:19
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634881/

[7634883] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_orch_cli}
time:   00:19:33
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634883/

[7634887] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_ca_signed_key}
time:   00:11:33
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634887/

[7634892] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:24:58
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634892/

[7634895] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_orch_cli_mon}
time:   00:24:46
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634895/

[7634896] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli}
time:   00:17:12
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634896/

[7634898] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final}
time:   00:13:10
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634898/

[7634903] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm_repos}
time:   00:06:11
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634903/

[7634907] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_extra_daemon_features}
time:   00:13:17
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634907/

[7634910] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_adoption}
time:   00:10:00
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634910/

[7634913] orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rotate-keys}
time:   00:23:20
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634913/

[7634916] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_host_drain}
time:   00:23:59
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634916/

[7634917] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:14:34
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634917/

[7634922] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}}
time:   00:13:27
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634922/

[7634924] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_cephadm_timeout}
time:   00:19:36
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634924/

[7634930] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_monitoring_stack_basic}
time:   00:18:59
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634930/

[7634934] orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:15:06
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634934/

[7634935] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_rgw_multisite}
time:   00:20:20
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634935/

[7634938] orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs}
time:   00:33:42
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634938/

[7634939] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli}
time:   00:15:00
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634939/

[7634940] orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli}
time:   00:13:54
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634940/

[7634941] orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi}
time:   00:24:24
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634941/

[7634943] orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final}
time:   00:11:24
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634943/

[7634946] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_set_mon_crush_locations}
time:   00:20:08
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634946/

[7634954] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_ca_signed_key}
time:   00:15:06
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634954/

[7634960] orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:24:42
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634960/

[7634965] orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rotate-keys}
time:   00:24:16
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634965/

[7634968] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:33:49
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634968/

[7634969] orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption}
time:   00:12:16
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634969/

[7634973] orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_extra_daemon_features}
time:   00:14:17
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634973/

[7634977] orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:15:44
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634977/

[7634981] orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_host_drain}
time:   00:17:13
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634981/

[7634983] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout}
time:   00:15:07
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634983/

[7634984] orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli}
time:   00:12:42
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634984/

[7634986] orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final}
time:   00:21:07
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634986/

[7634989] orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}}
time:   00:13:34
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634989/

[7634995] orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_monitoring_stack_basic}
time:   00:19:47
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634995/

[7634996] orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:36:16
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634996/

[7634997] orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli}
time:   00:13:24
info:   https://pulpito.ceph.com/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634997/