2024-03-20T22:32:59.720 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-20T22:32:59.720 INFO:teuthology.results:subset: '1/15'
2024-03-20T22:32:59.720 INFO:teuthology.results:seed: '4700'
2024-03-20T23:35:56.423 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-20T23:35:56.446 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-21T11:35:58.063 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2024-03-21T11:36:05.019 INFO:scrape:Found 93 jobs
2024-03-21T11:36:06.137 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.138 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.138 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.175 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.176 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.176 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.205 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.206 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.206 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.235 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.236 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:06.236 WARNING:scrape:Saw end of BT but not start
2024-03-21T11:36:21.212 INFO:scrape:Found 7 distinct failure reasons
2024-03-21T11:36:21.213 INFO:scrape:Failure: "2024-03-21T00:10:00.000152+0000 mon.a (mon.0) 1834 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log
2024-03-21T11:36:21.213 INFO:scrape:16 jobs: ['7612844', '7612785', '7612834', '7612753', '7612830', '7612821', '7612814', '7612764', '7612770', '7612784', '7612833', '7612786', '7612808', '7612771', '7612810', '7612794']
2024-03-21T11:36:21.213 INFO:scrape:suites intersection: []
2024-03-21T11:36:21.213 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/with-work/{0-distro/centos_9.stream', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'root}', 'start', 'start}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}']
2024-03-21T11:36:21.214 INFO:scrape:
2024-03-21T11:36:21.214 INFO:scrape:Dead: b'2024-03-20T23:56:36.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo nvme disconnect -n lv_2'
2024-03-21T11:36:21.214 INFO:scrape:1 jobs: ['7612840']
2024-03-21T11:36:21.214 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/basic', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream']
2024-03-21T11:36:21.214 INFO:scrape:
2024-03-21T11:36:21.214 INFO:scrape:Failure: "2024-03-20T23:53:22.858704+0000 mon.a (mon.0) 492 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2024-03-21T11:36:21.214 INFO:scrape:7 jobs: ['7612841', '7612777', '7612752', '7612773', '7612837', '7612819', '7612757']
2024-03-21T11:36:21.214 INFO:scrape:suites intersection: []
2024-03-21T11:36:21.214 INFO:scrape:suites union: ['1-bootstrap/17.2.0', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-inline/no', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-03-21T11:36:21.215 INFO:scrape:
2024-03-21T11:36:21.215 INFO:scrape:Dead: b'2024-03-21T11:24:55.271 DEBUG:teuthology.exit:Finished running handlers'
2024-03-21T11:36:21.215 INFO:scrape:1 jobs: ['7612818']
2024-03-21T11:36:21.215 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-keepalive-only', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream']
2024-03-21T11:36:21.215 INFO:scrape:
2024-03-21T11:36:21.215 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf39634006a81dfc721a310ec1ab21686764f4a5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
2024-03-21T11:36:21.215 INFO:scrape:1 jobs: ['7612820']
2024-03-21T11:36:21.215 INFO:scrape:suites: ['1-start', '2-thrash', '3-tasks/rados_api_tests', 'fixed-2', 'msgr/async-v1only', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'root}']
2024-03-21T11:36:21.215 INFO:scrape:
2024-03-21T11:36:21.216 INFO:scrape:Failure: Command failed on smithi121 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 06a25d32-e713-11ee-95c9-87774f69a715 -e sha1=bf39634006a81dfc721a310ec1ab21686764f4a5 -- bash -c 'ceph orch ps'"
2024-03-21T11:36:21.216 INFO:scrape:4 jobs: ['7612824', '7612789', '7612799', '7612803']
2024-03-21T11:36:21.216 INFO:scrape:suites intersection: []
2024-03-21T11:36:21.216 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-repo_digest/defaut', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-03-21T11:36:21.216 INFO:scrape:
2024-03-21T11:36:21.216 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf39634006a81dfc721a310ec1ab21686764f4a5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2024-03-21T11:36:21.216 INFO:scrape:1 jobs: ['7612790']
2024-03-21T11:36:21.216 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}']
2024-03-21T11:36:21.217 INFO:scrape:
2024-03-21T11:36:21.220 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/
failed:  29
dead:    1
running: 1
waiting: 0
queued:  0
passed:  62


Fail
=================================================================
[7612752]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:48:13
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612752/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612752/

    "2024-03-20T22:52:41.381883+0000 mon.a (mon.0) 168 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7612753]  orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:36:13
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612753/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612753/

    "2024-03-20T23:10:00.000150+0000 mon.a (mon.0) 1551 : cluster 3 [WRN]
    OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

[7612757]  orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:34:32
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612757/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612757/

    "2024-03-20T23:06:12.155220+0000 mon.smithi045 (mon.0) 856 : cluster [WRN]
    Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in
    cluster log

[7612764]  orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:11:30
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612764/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612764/

    "2024-03-20T23:00:18.211358+0000 mon.a (mon.0) 190 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7612770]  orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   01:03:59
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612770/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612770/

    "2024-03-20T23:11:55.378838+0000 mon.a (mon.0) 177 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7612771]  orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:39:07
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612771/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612771/

    "2024-03-20T23:12:57.161584+0000 mon.a (mon.0) 182 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7612773]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:31:32
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612773/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612773/

    "1710976609.7666574 mon.smithi102 (mon.0) 680 : cluster [WRN] Health check
    failed: insufficient standby MDS daemons available
    (MDS_INSUFFICIENT_STANDBY)" in cluster log

[7612777]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:41:45
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612777/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612777/

    "2024-03-20T23:12:03.312787+0000 mon.a (mon.0) 207 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7612784]  orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:12:14
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612784/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612784/

    "2024-03-20T23:18:39.865505+0000 mon.a (mon.0) 246 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7612785]  orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:56:23
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612785/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612785/

    "2024-03-20T23:30:00.000145+0000 mon.a (mon.0) 1021 : cluster 3 [WRN]
    OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

[7612786]  orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   01:01:32
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612786/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612786/

    "2024-03-20T23:40:00.000207+0000 mon.a (mon.0) 2318 : cluster 3 [WRN]
    POOL_APP_NOT_ENABLED: 7 pool(s) do not have an application enabled" in
    cluster log

[7612789]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   03:36:49
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612789/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612789/
sentry: https://sentry.ceph.com/organizations/ceph/?query=377c4817d20340bdaebbd7dca75aefa2

    Command failed on smithi087 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    8b6072be-e710-11ee-95c9-87774f69a715 -e
    sha1=bf39634006a81dfc721a310ec1ab21686764f4a5 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7612790]  orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:17:36
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612790/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612790/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ad5830e2087f48e098223885027ff035

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi118 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bf39634006a81dfc721a310ec1ab21686764f4a5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[7612794]  orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:30:56
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612794/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612794/

    "2024-03-20T23:40:00.000116+0000 mon.a (mon.0) 1433 : cluster 3 [WRN]
    OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

[7612799]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:43:42
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612799/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612799/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1892d8d90f834c4d84e2b23b6f4ad67d

    Command failed on smithi163 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    e2ddcf28-e710-11ee-95c9-87774f69a715 -e
    sha1=bf39634006a81dfc721a310ec1ab21686764f4a5 -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 7\'"\'"\'\''

[7612803]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:20
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612803/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612803/
sentry: https://sentry.ceph.com/organizations/ceph/?query=623fd222a5b54f7a8dcc51147431b940

    Command failed on smithi028 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    3a5a63a6-e711-11ee-95c9-87774f69a715 -e
    sha1=bf39634006a81dfc721a310ec1ab21686764f4a5 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

[7612808]  orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:12:05
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612808/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612808/

    "2024-03-20T23:31:55.644724+0000 mon.a (mon.0) 246 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7612810]  orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:52:27
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612810/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612810/

    "2024-03-21T00:10:00.009882+0000 mon.a (mon.0) 2368 : cluster 3 [WRN]
    OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

[7612814]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
-----------------------------------------------------------------
time:   00:16:42
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612814/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612814/

    "2024-03-20T23:40:00.000232+0000 mon.smithi050 (mon.0) 865 : cluster 3
    [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 2 daemon(s)" in cluster
    log

[7612819]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:38:02
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612819/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612819/

    "2024-03-20T23:51:49.355639+0000 mon.a (mon.0) 960 : cluster [WRN] Health
    check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in
    cluster log

[7612820]  orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   01:04:01
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612820/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612820/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ba147d3ae28940dba36702589ac9053b

    Command failed (workunit test rados/test.sh) on smithi005 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bf39634006a81dfc721a310ec1ab21686764f4a5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

[7612821]  orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   01:01:29
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612821/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612821/

    "2024-03-20T23:41:38.510734+0000 mon.a (mon.0) 178 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7612824]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:52:07
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612824/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612824/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6cf0d2e9e40a44b79953baf684c3e690

    Command failed on smithi121 with status 1: "sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy
    shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    06a25d32-e713-11ee-95c9-87774f69a715 -e
    sha1=bf39634006a81dfc721a310ec1ab21686764f4a5 -- bash -c 'ceph orch ps'"

[7612830]  orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:11:23
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612830/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612830/

    "2024-03-20T23:42:28.355493+0000 mon.a (mon.0) 188 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

[7612833]  orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   01:00:00
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612833/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612833/

    "2024-03-21T00:10:00.000139+0000 mon.a (mon.0) 1990 : cluster 3 [WRN]
    OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

[7612834]  orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:23:14
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612834/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612834/

    "2024-03-20T23:44:29.039180+0000 mon.a (mon.0) 177 : cluster 3 [WRN]
    MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

[7612837]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:33:39
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612837/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612837/

    "1710979258.2632377 mon.smithi129 (mon.0) 885 : cluster [WRN] Health check
    failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

[7612841]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:27:41
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612841/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612841/

    "2024-03-20T23:53:22.858704+0000 mon.a (mon.0) 492 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[7612844]  orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:30:26
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612844/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612844/

    "2024-03-21T00:10:00.000152+0000 mon.a (mon.0) 1834 : cluster 3 [WRN]
    PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't
    resolve itself): 1 pg backfill_toofull" in cluster log



Dead
=================================================================
[7612818]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612818/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612818/

    hit max job timeout



Running
=================================================================
[7612840] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final}
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612840/



Pass
=================================================================
[7612754] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python}
time:   00:22:41
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612754/

[7612755] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}}
time:   00:13:19
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612755/

[7612756] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:16:22
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612756/

[7612758] orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs}
time:   00:33:11
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612758/

[7612759] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli}
time:   00:11:46
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612759/

[7612760] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:19:33
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612760/

[7612761] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi}
time:   00:26:03
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612761/

[7612762] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:14:30
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612762/

[7612763] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final}
time:   00:11:23
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612763/

[7612765] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
time:   00:21:23
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612765/

[7612766] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic}
time:   00:25:49
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612766/

[7612767] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:14:10
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612767/

[7612768] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon}
time:   00:24:47
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612768/

[7612769] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:14:03
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612769/

[7612772] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite}
time:   00:13:28
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612772/

[7612774] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:14:29
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612774/

[7612775] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final}
time:   00:20:04
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612775/

[7612776] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:17:35
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612776/

[7612778] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations}
time:   00:16:05
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612778/

[7612779] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:15:48
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612779/

[7612780] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption}
time:   00:12:33
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612780/

[7612781] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:13:05
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612781/

[7612782] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:26:22
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612782/

[7612783] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:18:54
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612783/

[7612787] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key}
time:   00:11:36
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612787/

[7612788] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:13:17
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612788/

[7612791] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:14:10
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612791/

[7612792] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:21:31
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612792/

[7612793] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:19:33
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612793/

[7612795] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python}
time:   00:22:28
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612795/

[7612796] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos}
time:   00:06:25
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612796/

[7612797] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout}
time:   00:14:35
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612797/

[7612798] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:14:12
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612798/

[7612800] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
time:   00:15:24
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612800/

[7612801] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features}
time:   00:13:55
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612801/

[7612802] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:22:13
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612802/

[7612804] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli}
time:   00:11:46
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612804/

[7612805] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:13:26
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612805/

[7612806] orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:15:26
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612806/

[7612807] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final}
time:   00:12:28
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612807/

[7612809] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:15:26
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612809/

[7612811] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys}
time:   00:23:10
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612811/

[7612812] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain}
time:   00:22:40
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612812/

[7612813] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli}
time:   00:13:17
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612813/

[7612815] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}}
time:   00:13:45
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612815/

[7612816] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:23:39
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612816/

[7612817] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:13:08
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612817/

[7612822] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic}
time:   00:19:02
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612822/

[7612823] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:15:04
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612823/

[7612825] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon}
time:   00:37:22
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612825/

[7612826] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:19:49
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612826/

[7612827] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite}
time:   00:20:18
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612827/

[7612828] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:21:24
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612828/

[7612829] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:14:47
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612829/

[7612831] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final}
time:   00:13:48
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612831/

[7612832] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:18:11
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612832/

[7612835] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations}
time:   00:15:06
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612835/

[7612836] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:21:17
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612836/

[7612838] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_adoption}
time:   00:10:02
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612838/

[7612839] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:13:03
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612839/

[7612842] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key}
time:   00:11:38
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612842/

[7612843] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:13:54
info:   https://pulpito.ceph.com/adking-2024-03-20_17:29:20-orch:cephadm-wip-adk4-testing-2024-03-20-0737-squid-distro-default-smithi/7612843/