2024-02-27T05:41:42.220 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-27T05:41:42.220 INFO:teuthology.results:subset: '1/14'
2024-02-27T05:41:42.220 INFO:teuthology.results:seed: '3790'
2024-02-27T07:44:44.828 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-27T07:44:44.854 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-27T18:49:30.562 INFO:teuthology.results:Tests finished! gathering results...
2024-02-27T18:49:37.703 INFO:scrape:Found 95 jobs
2024-02-27T18:49:43.214 WARNING:scrape:Saw end of BT but not start
2024-02-27T18:49:43.214 WARNING:scrape:Saw end of BT but not start
2024-02-27T18:49:43.215 WARNING:scrape:Saw end of BT but not start
2024-02-27T18:49:51.628 WARNING:scrape:Saw end of BT but not start
2024-02-27T18:49:51.629 WARNING:scrape:Saw end of BT but not start
2024-02-27T18:49:51.629 WARNING:scrape:Saw end of BT but not start
2024-02-27T18:49:51.638 INFO:scrape:Found 8 distinct failure reasons
2024-02-27T18:49:51.638 INFO:scrape:Failure: "2024-02-27T07:07:56.784412+0000 mon.a (mon.0) 172 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log
2024-02-27T18:49:51.638 INFO:scrape:12 jobs: ['7574517', '7574507', '7574552', '7574520', '7574463', '7574460', '7574538', '7574503', '7574519', '7574477', '7574497', '7574539']
2024-02-27T18:49:51.638 INFO:scrape:suites intersection: []
2024-02-27T18:49:51.638 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-start', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/smoke/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'root}', 'start', 'start}', 'task/test_host_drain}', 'tasks/rados_api_tests}']
2024-02-27T18:49:51.639 INFO:scrape:
2024-02-27T18:49:51.639 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi094 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=635ee6850c57c98cc6ba932cade45f9cc015aa32 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'
2024-02-27T18:49:51.639 INFO:scrape:1 jobs: ['7574549']
2024-02-27T18:49:51.639 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}']
2024-02-27T18:49:51.639 INFO:scrape:
2024-02-27T18:49:51.639 INFO:scrape:Failure: Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f869c4fa-d536-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''
2024-02-27T18:49:51.639 INFO:scrape:2 jobs: ['7574462', '7574528']
2024-02-27T18:49:51.639 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}']
2024-02-27T18:49:51.640 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}']
2024-02-27T18:49:51.640 INFO:scrape:
2024-02-27T18:49:51.640 INFO:scrape:Dead: b'2024-02-27T18:46:48.587 DEBUG:teuthology.exit:Finished running handlers'
2024-02-27T18:49:51.640 INFO:scrape:1 jobs: ['7574511']
2024-02-27T18:49:51.640 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'task/test_iscsi_container/{centos_9.stream', 'test_iscsi_container}}']
2024-02-27T18:49:51.640 INFO:scrape:
2024-02-27T18:49:51.640 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi006 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=635ee6850c57c98cc6ba932cade45f9cc015aa32 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2024-02-27T18:49:51.640 INFO:scrape:1 jobs: ['7574545']
2024-02-27T18:49:51.640 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-02-27T18:49:51.640 INFO:scrape:
2024-02-27T18:49:51.641 INFO:scrape:Failure: Test failure: test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS)
2024-02-27T18:49:51.641 INFO:scrape:1 jobs: ['7574464']
2024-02-27T18:49:51.641 INFO:scrape:suites: ['conf/{client', 'mds', 'mon', 'orch:cephadm/nfs/{cluster/{1-node}', 'osd}', 'overrides/ignorelist_health', 'supported-random-distros$/{ubuntu_latest}', 'tasks/nfs}']
2024-02-27T18:49:51.641 INFO:scrape:
2024-02-27T18:49:51.641 INFO:scrape:Failure: "2024-02-27T07:20:00.000137+0000 mon.smithi089 (mon.0) 481 : cluster 3 [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid" in cluster log
2024-02-27T18:49:51.641 INFO:scrape:1 jobs: ['7574516']
2024-02-27T18:49:51.641 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2024-02-27T18:49:51.641 INFO:scrape:
2024-02-27T18:49:51.641 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-02-27T18:49:51.641 INFO:scrape:3 jobs: ['7574487', '7574505', '7574534']
2024-02-27T18:49:51.642 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}']
2024-02-27T18:49:51.642 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}']
2024-02-27T18:49:51.642 INFO:scrape:
2024-02-27T18:49:51.645 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/
failed:  21
dead:    1
running: 0
waiting: 0
queued:  0
passed:  73


Fail
=================================================================
[7574460]  orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   01:00:58
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574460/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574460/

    "2024-02-27T06:40:00.000211+0000 mon.a (mon.0) 1620 : cluster 3 [WRN]
    POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in
    cluster log

[7574462]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:25:44
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574462/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574462/
sentry: https://sentry.ceph.com/organizations/ceph/?query=57a8e1d21edd4731b86c42d7284d5b28

    Command failed on smithi057 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell
    -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    f869c4fa-d536-11ee-95c3-87774f69a715 -e
    sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7574463]  orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:35:10
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463/

    "2024-02-27T06:21:27.967462+0000 mon.smithi131 (mon.0) 743 : cluster [WRN]
    Health check failed: Degraded data redundancy: 1/723 objects degraded
    (0.138%), 1 pg degraded (PG_DEGRADED)" in cluster log

[7574464]  orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs}
-----------------------------------------------------------------
time:   00:22:23
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574464/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574464/
sentry: https://sentry.ceph.com/organizations/ceph/?query=120eb784246346d191a71c5736c038f4

    Test failure: test_create_and_delete_export (tasks.cephfs.test_nfs.TestNFS)

[7574477]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:39:40
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574477/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574477/

    "2024-02-27T06:25:44.349078+0000 mon.a (mon.0) 318 : cluster [WRN] Health
    check failed: 1 stray daemon(s) not managed by cephadm
    (CEPHADM_STRAY_DAEMON)" in cluster log

[7574487]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:07:32
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574487/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574487/

    reached maximum tries (51) after waiting for 300 seconds

[7574497]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:51:35
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574497/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574497/

    "2024-02-27T06:47:33.821991+0000 mon.a (mon.0) 200 : cluster [WRN] Health
    detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log

[7574503]  orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:15:16
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574503/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574503/

    "2024-02-27T07:00:00.000189+0000 mon.a (mon.0) 1029 : cluster 3 [WRN]
    CEPHADM_AGENT_DOWN: 2 Cephadm Agent(s) are not reporting. Hosts may be
    offline" in cluster log

[7574505]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:39:47
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574505/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574505/

    reached maximum tries (51) after waiting for 300 seconds

[7574507]  orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain}
-----------------------------------------------------------------
time:   00:23:06
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574507/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574507/

    "2024-02-27T07:10:00.000216+0000 mon.a (mon.0) 706 : cluster 3 [WRN]
    CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed by
    cephadm" in cluster log

[7574516]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:31:53
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574516/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574516/

    "2024-02-27T07:20:00.000137+0000 mon.smithi089 (mon.0) 481 : cluster 3
    [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but
    require_osd_release < squid" in cluster log

[7574517]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:37:39
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574517/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574517/

    "2024-02-27T07:07:56.784412+0000 mon.a (mon.0) 172 : cluster [WRN] Health
    detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log

[7574519]  orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   01:04:20
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574519/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574519/

    "2024-02-27T07:30:00.003362+0000 mon.a (mon.0) 2063 : cluster 3 [WRN]
    OSD_DOWN: 1 osds down" in cluster log

[7574520]  orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   01:01:24
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574520/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574520/

    "2024-02-27T07:30:00.217333+0000 mon.a (mon.0) 1268 : cluster 3 [WRN]
    CEPHADM_STRAY_DAEMON: 2 stray daemon(s) not managed by cephadm" in cluster
    log

[7574528]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:13
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574528/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574528/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aae5f84d9dc84b71b7675459523065aa

    Command failed on smithi077 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell
    -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    1a691c5e-d541-11ee-95c3-87774f69a715 -e
    sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c \'ceph versions |
    jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

[7574534]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:39:50
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574534/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574534/

    reached maximum tries (51) after waiting for 300 seconds

[7574538]  orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:56:43
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574538/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574538/

    "2024-02-27T07:50:00.000142+0000 mon.a (mon.0) 896 : cluster 3 [WRN]
    PG_DEGRADED: Degraded data redundancy: 3284/201426 objects degraded
    (1.630%), 2 pgs degraded" in cluster log

[7574539]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:28:19
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574539/

    "2024-02-27T07:51:40.059228+0000 mon.a (mon.0) 948 : cluster [WRN] Health
    check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in
    cluster log

[7574545]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:26:52
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574545/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574545/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eab2018196d445fc84eb61134a4d3b71

    Command failed (workunit test suites/fsstress.sh) on smithi006 with status
    126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=635ee6850c57c98cc6ba932cade45f9cc015aa32
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7574549]  orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos}
-----------------------------------------------------------------
time:   00:08:39
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574549/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574549/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e25c4d21b0c04ee689a91d7e810a5594

    Command failed (workunit test cephadm/test_repos.sh) on smithi094 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=635ee6850c57c98cc6ba932cade45f9cc015aa32
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

[7574552]  orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:31:41
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574552/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574552/

    "2024-02-27T08:20:00.000210+0000 mon.a (mon.0) 1841 : cluster 3 [WRN]
    OSD_DOWN: 1 osds down" in cluster log



Dead
=================================================================
[7574511]  orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574511/
log:    http://qa-proxy.ceph.com/teuthology/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574511/

    hit max job timeout



Pass
=================================================================
[7574458] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:14:59
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574458/

[7574459] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:21:40
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574459/

[7574461] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic}
time:   00:25:11
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574461/

[7574465] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli}
time:   00:13:50
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574465/

[7574466] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi}
time:   00:30:33
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574466/

[7574467] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:18:04
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574467/

[7574468] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final}
time:   00:18:16
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574468/

[7574469] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
time:   00:12:00
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574469/

[7574470] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon}
time:   00:25:29
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574470/

[7574471] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys}
time:   00:38:42
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574471/

[7574472] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite}
time:   00:14:57
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574472/

[7574473] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:17:10
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574473/

[7574474] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:19:06
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574474/

[7574475] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
time:   00:21:40
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574475/

[7574476] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:32:47
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574476/

[7574478] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations}
time:   00:16:59
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574478/

[7574479] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:14:49
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574479/

[7574480] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption}
time:   00:12:26
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574480/

[7574481] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:25:17
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574481/

[7574482] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:14:05
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574482/

[7574483] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
time:   00:33:18
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574483/

[7574484] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests}
time:   01:01:14
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574484/

[7574485] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key}
time:   00:12:03
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574485/

[7574486] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:14:53
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574486/

[7574488] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final}
time:   00:19:54
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574488/

[7574489] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm}
time:   00:19:53
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574489/

[7574490] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:18:07
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574490/

[7574491] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
time:   00:12:04
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574491/

[7574492] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final}
time:   00:16:20
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574492/

[7574493] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python}
time:   00:24:46
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574493/

[7574494] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos}
time:   00:06:49
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574494/

[7574495] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:32:51
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574495/

[7574496] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout}
time:   00:15:03
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574496/

[7574498] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all}
time:   00:13:32
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574498/

[7574499] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final}
time:   00:19:13
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574499/

[7574500] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root}
time:   00:40:09
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574500/

[7574501] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features}
time:   00:15:07
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574501/

[7574502] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:14:28
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574502/

[7574504] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final}
time:   00:14:23
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574504/

[7574506] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys}
time:   00:28:02
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574506/

[7574508] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli}
time:   00:13:24
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574508/

[7574509] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final}
time:   00:22:22
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574509/

[7574510] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
time:   00:19:22
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574510/

[7574512] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli}
time:   00:13:09
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574512/

[7574513] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:14:26
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574513/

[7574514] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final}
time:   00:12:51
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574514/

[7574515] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:12:38
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574515/

[7574518] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
time:   00:16:29
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574518/

[7574521] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic}
time:   00:20:12
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574521/

[7574522] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
time:   00:21:55
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574522/

[7574523] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag}
time:   00:13:24
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574523/

[7574524] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon}
time:   00:37:16
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574524/

[7574525] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite}
time:   00:19:55
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574525/

[7574526] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
time:   00:15:44
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574526/

[7574527] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:14:59
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574527/

[7574529] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
time:   00:17:21
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574529/

[7574530] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
time:   00:22:48
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574530/

[7574531] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations}
time:   00:17:03
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574531/

[7574532] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
time:   00:24:04
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574532/

[7574533] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait}
time:   00:13:22
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574533/

[7574535] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_adoption}
time:   00:09:50
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574535/

[7574536] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
time:   00:15:03
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574536/

[7574537] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
time:   00:11:57
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574537/

[7574540] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key}
time:   00:11:40
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574540/

[7574541] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final}
time:   00:14:07
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574541/

[7574542] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys}
time:   00:24:36
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574542/

[7574543] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm}
time:   00:17:37
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574543/

[7574544] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:19:43
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574544/

[7574546] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
time:   00:20:49
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574546/

[7574547] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start}
time:   00:25:31
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574547/

[7574548] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final}
time:   00:14:21
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574548/

[7574550] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_cephadm_timeout}
time:   00:14:15
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574550/

[7574551] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:18:48
info:   https://pulpito.ceph.com/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574551/