2024-04-08T19:18:07.716 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-08T19:18:07.716 INFO:teuthology.results:subset: '1/20'
2024-04-08T19:18:07.716 INFO:teuthology.results:seed: '2521'
2024-04-08T20:42:32.998 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-08T20:42:33.021 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-09T08:43:32.711 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2024-04-09T08:43:39.542 INFO:scrape:Found 89 jobs
2024-04-09T08:43:42.006 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645905/teuthology.log
2024-04-09T08:44:00.794 INFO:scrape:Found 7 distinct failure reasons
2024-04-09T08:44:00.795 INFO:scrape:Failure: "2024-04-08T20:29:25.593601+0000 mon.smithi092 (mon.0) 598 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
2024-04-09T08:44:00.795 INFO:scrape:63 jobs
2024-04-09T08:44:00.795 INFO:scrape:suites intersection: []
2024-04-09T08:44:00.796 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '2-nfs', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/basic', '2-services/iscsi', '2-services/jaeger', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-thrash', '3-final}', '3-inline/no', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'base/install', 'centos_9.stream', 'cluster/{fixed-3', 'conf/{client', 'conf/{disable-pool-app}', 'fail_fs/no', 'fixed-2', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/nfs/{cluster/{1-node}', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04', 'orch:cephadm/osds/{0-distro/centos_9.stream', 'orch:cephadm/osds/{0-distro/centos_9.stream_runc', 'orch:cephadm/osds/{0-distro/ubuntu_22.04', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smb/{0-distro/centos_9.stream', 'orch:cephadm/smb/{0-distro/centos_9.stream_runc', 'orch:cephadm/smb/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/centos_9.stream', 'orch:cephadm/smoke/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/ubuntu_22.04', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04', 'orch:cephadm/with-work/{0-distro/centos_9.stream', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'roles', 'root}', 'start', 'start}', 'supported-random-distros$/{ubuntu_latest}', 'syntax}', 'task/test_cephadm_timeout}', 'task/test_host_drain}', 'task/test_iscsi_container/{centos_9.stream', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/deploy_smb_basic}', 'tasks/deploy_smb_domain}', 'tasks/deploy_smb_mgr_domain}', 'tasks/nfs}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'tasks/{0-from/quincy', 'test_iscsi_container}}', 'workloads/cephadm_iscsi}']
2024-04-09T08:44:00.796 INFO:scrape:
2024-04-09T08:44:00.796 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-04-09T08:44:00.796 INFO:scrape:5 jobs: ['7645927', '7645907', '7645963', '7645884', '7645956']
2024-04-09T08:44:00.796 INFO:scrape:suites intersection: []
2024-04-09T08:44:00.796 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smb/{0-distro/centos_9.stream_runc', 'orch:cephadm/smb/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/deploy_smb_mgr_res_dom}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.0}']
2024-04-09T08:44:00.796 INFO:scrape:
2024-04-09T08:44:00.796 INFO:scrape:Dead: None
2024-04-09T08:44:00.797 INFO:scrape:2 jobs: ['7645905', '7645904']
2024-04-09T08:44:00.797 INFO:scrape:
2024-04-09T08:44:00.797 INFO:scrape:Dead: b'2024-04-09T08:41:39.141 DEBUG:teuthology.exit:Finished running handlers'
2024-04-09T08:44:00.797 INFO:scrape:3 jobs: ['7645959', '7645886', '7645894']
2024-04-09T08:44:00.797 INFO:scrape:suites intersection: ['0-nvme-loop']
2024-04-09T08:44:00.797 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '3-final}', 'agent/off', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/centos_9.stream_runc', 'start}']
2024-04-09T08:44:00.797 INFO:scrape:
2024-04-09T08:44:00.797 INFO:scrape:Failure: Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4432bbac-f5ea-11ee-b649-cb9ed24678a4 -e sha1=fb6f9800392d73c10dc835c8e8e86c5884686270 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''
2024-04-09T08:44:00.797 INFO:scrape:1 jobs: ['7645971']
2024-04-09T08:44:00.797 INFO:scrape:suites: ['2-repo_digest/defaut', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/on', 'mon_election/classic}', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream']
2024-04-09T08:44:00.798 INFO:scrape:
2024-04-09T08:44:00.798 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fb6f9800392d73c10dc835c8e8e86c5884686270 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2024-04-09T08:44:00.798 INFO:scrape:1 jobs: ['7645960']
2024-04-09T08:44:00.798 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}']
2024-04-09T08:44:00.798 INFO:scrape:
2024-04-09T08:44:00.798 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
2024-04-09T08:44:00.798 INFO:scrape:1 jobs: ['7645901']
2024-04-09T08:44:00.798 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}']
2024-04-09T08:44:00.798 INFO:scrape:
2024-04-09T08:44:00.801 INFO:teuthology.results:Sending results to jmulligan@redhat.com: Test Run: phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/
failed:  71
dead:    4
running: 1
waiting: 0
queued:  0
passed:  13


Fail
=================================================================
[7645883]  orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag}
-----------------------------------------------------------------
time:   00:13:38
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645883/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645883/

    "2024-04-08T19:42:23.881306+0000 mon.smithi063 (mon.0) 782 : cluster [WRN]
    Health check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645884]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:37:28
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645884/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645884/

    reached maximum tries (51) after waiting for 300 seconds

[7645885]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
-----------------------------------------------------------------
time:   00:57:25
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645885/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645885/

    "2024-04-08T19:38:37.950462+0000 mon.a (mon.0) 172 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7645887]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
-----------------------------------------------------------------
time:   00:21:56
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645887/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645887/

    "2024-04-08T19:45:05.626428+0000 mon.smithi103 (mon.0) 686 : cluster [WRN]
    Health check failed: Degraded data redundancy: 2/6 objects degraded
    (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log

[7645888]  orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}
-----------------------------------------------------------------
time:   00:34:39
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645888/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645888/

    "2024-04-08T19:43:47.357875+0000 mon.smithi113 (mon.0) 494 : cluster [WRN]
    Health check failed: Reduced data availability: 1 pg peering
    (PG_AVAILABILITY)" in cluster log

[7645889]  orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs}
-----------------------------------------------------------------
time:   00:46:38
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645889/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645889/

    "2024-04-08T19:50:02.911275+0000 mon.a (mon.0) 321 : cluster [WRN] Health
    check failed: no active mgr (MGR_DOWN)" in cluster log

[7645890]  orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli}
-----------------------------------------------------------------
time:   00:14:14
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645890/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645890/

    "2024-04-08T19:45:58.885355+0000 mon.a (mon.0) 463 : cluster [WRN] Health
    check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645892]  orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi}
-----------------------------------------------------------------
time:   00:27:56
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645892/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645892/

    "2024-04-08T19:45:02.875015+0000 mon.a (mon.0) 207 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7645893]  orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_domain}
-----------------------------------------------------------------
time:   00:11:42
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645893/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645893/

    "2024-04-08T19:46:13.047870+0000 mon.a (mon.0) 251 : cluster [WRN] Health
    check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645896]  orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:11:59
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645896/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645896/

    "2024-04-08T19:50:20.943896+0000 mon.a (mon.0) 207 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7645898]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
-----------------------------------------------------------------
time:   00:16:43
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645898/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645898/

    "2024-04-08T19:55:54.106915+0000 mon.smithi148 (mon.0) 777 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645899]  orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait}
-----------------------------------------------------------------
time:   00:18:48
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645899/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645899/

    "2024-04-08T20:03:17.116477+0000 mon.smithi133 (mon.0) 764 : cluster [WRN]
    Health check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645900]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:22:36
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645900/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645900/

    "2024-04-08T20:07:34.828646+0000 mon.smithi049 (mon.0) 770 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645901]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   03:40:02
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645901/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645901/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9790678fb09346ae9e7df84b31c0838e

    Command failed (workunit test suites/fsstress.sh) on smithi087 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=fb6f9800392d73c10dc835c8e8e86c5884686270
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[7645902]  orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   00:31:28
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645902/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645902/

    "2024-04-08T20:04:33.431512+0000 mon.a (mon.0) 851 : cluster [WRN] Health
    check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log

[7645903]  orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:37:10
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645903/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645903/

    "2024-04-08T20:07:37.534095+0000 mon.a (mon.0) 1169 : cluster [WRN] Health
    check failed: 2 stray daemon(s) not managed by cephadm
    (CEPHADM_STRAY_DAEMON)" in cluster log

[7645906]  orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   00:27:48
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645906/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645906/

    "2024-04-08T20:08:29.633886+0000 mon.a (mon.0) 271 : cluster [WRN] Health
    detail: HEALTH_WARN 1/4 mons down, quorum a,e,c" in cluster log

[7645907]  orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_res_dom}
-----------------------------------------------------------------
time:   00:35:39
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645907/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645907/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2501b5a8c9f64ad59c7bc35313999b02

    reached maximum tries (301) after waiting for 300 seconds

[7645909]  orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:26:25
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645909/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645909/

    "2024-04-08T20:06:12.353635+0000 mon.a (mon.0) 395 : cluster [WRN] Health
    check failed: 1 stray daemon(s) not managed by cephadm
    (CEPHADM_STRAY_DAEMON)" in cluster log

[7645910]  orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
-----------------------------------------------------------------
time:   00:14:59
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645910/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645910/

    "2024-04-08T20:05:17.823802+0000 mon.smithi055 (mon.0) 782 : cluster [WRN]
    Health check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645911]  orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain}
-----------------------------------------------------------------
time:   00:16:14
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645911/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645911/

    "2024-04-08T20:07:08.493633+0000 mon.a (mon.0) 615 : cluster [WRN] Health
    check failed: 1 stray daemon(s) not managed by cephadm
    (CEPHADM_STRAY_DAEMON)" in cluster log

[7645912]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final}
-----------------------------------------------------------------
time:   00:20:46
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645912/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645912/

    "2024-04-08T20:13:57.834835+0000 mon.smithi105 (mon.0) 760 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645913]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:33:01
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645913/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645913/

    "1712607336.505119 mon.smithi026 (mon.0) 954 : cluster [WRN] Health check
    failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

[7645915]  orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:12:16
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645915/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645915/

    "2024-04-08T20:11:33.141452+0000 mon.a (mon.0) 386 : cluster [WRN] Health
    check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in
    cluster log

[7645916]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic}
-----------------------------------------------------------------
time:   00:47:54
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645916/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645916/

    "2024-04-08T20:09:56.424693+0000 mon.a (mon.0) 168 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7645917]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final}
-----------------------------------------------------------------
time:   00:15:50
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645917/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645917/

    "2024-04-08T20:16:16.295858+0000 mon.smithi050 (mon.0) 795 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645918]  orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:35:37
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645918/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645918/

    "2024-04-08T20:18:20.873374+0000 mon.a (mon.0) 745 : cluster [WRN] Health
    check failed: Degraded data redundancy: 3/594 objects degraded (0.505%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[7645920]  orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}}
-----------------------------------------------------------------
time:   00:14:35
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645920/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645920/

    "2024-04-08T20:20:32.437970+0000 mon.a (mon.0) 299 : cluster [WRN] Health
    check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645922]  orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all}
-----------------------------------------------------------------
time:   00:12:56
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645922/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645922/

    "2024-04-08T20:27:43.212309+0000 mon.smithi031 (mon.0) 795 : cluster [WRN]
    Health check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645923]  orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic}
-----------------------------------------------------------------
time:   00:18:51
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645923/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645923/

    "2024-04-08T20:33:42.509787+0000 mon.a (mon.0) 246 : cluster [WRN] Health
    check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645924]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final}
-----------------------------------------------------------------
time:   00:18:55
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645924/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645924/

    "2024-04-08T20:31:27.585717+0000 mon.smithi028 (mon.0) 674 : cluster [WRN]
    Health check failed: Degraded data redundancy: 2/6 objects degraded
    (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log

[7645926]  orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic}
-----------------------------------------------------------------
time:   00:25:18
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645926/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645926/

    "2024-04-08T20:35:25.765937+0000 mon.a (mon.0) 432 : cluster [WRN] Health
    check failed: 1 stray daemon(s) not managed by cephadm
    (CEPHADM_STRAY_DAEMON)" in cluster log

[7645927]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:07:46
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645927/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645927/

    reached maximum tries (51) after waiting for 300 seconds

[7645928]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final}
-----------------------------------------------------------------
time:   00:14:12
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645928/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645928/

    "2024-04-08T20:29:25.593601+0000 mon.smithi092 (mon.0) 598 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645929]  orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:15:25
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645929/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645929/

    "2024-04-08T20:27:52.185874+0000 mon.a (mon.0) 420 : cluster [WRN] Health
    check failed: 1 stray daemon(s) not managed by cephadm
    (CEPHADM_STRAY_DAEMON)" in cluster log

[7645930]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final}
-----------------------------------------------------------------
time:   00:21:14
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645930/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645930/

    "2024-04-08T20:33:50.744507+0000 mon.smithi055 (mon.0) 591 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645931]  orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
-----------------------------------------------------------------
time:   00:19:40
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645931/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645931/

    "2024-04-08T20:37:06.674123+0000 mon.smithi017 (mon.0) 764 : cluster [WRN]
    Health check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645932]  orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
-----------------------------------------------------------------
time:   00:46:59
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645932/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645932/

    "2024-04-08T20:33:41.493113+0000 mon.a (mon.0) 199 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

[7645933]  orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys}
-----------------------------------------------------------------
time:   00:39:06
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645933/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645933/

    "2024-04-08T20:34:52.995119+0000 mon.a (mon.0) 200 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

[7645934]  orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite}
-----------------------------------------------------------------
time:   00:14:23
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645934/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645934/

    "2024-04-08T20:36:51.971072+0000 mon.a (mon.0) 209 : cluster [WRN] Health
    detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log

[7645935]  orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout}
-----------------------------------------------------------------
time:   00:14:32
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645935/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645935/

    "2024-04-08T20:38:47.087630+0000 mon.a (mon.0) 210 : cluster [WRN] Health
    check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)"
    in cluster log

[7645937]  orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain}
-----------------------------------------------------------------
time:   00:11:53
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645937/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645937/

    "2024-04-08T20:37:38.621032+0000 mon.a (mon.0) 242 : cluster [WRN] Health
    check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645940]  orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:12:19
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645940/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645940/

    "2024-04-08T20:41:34.393305+0000 mon.a (mon.0) 265 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7645941]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final}
-----------------------------------------------------------------
time:   00:15:37
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645941/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645941/

    "2024-04-08T20:45:27.994680+0000 mon.smithi106 (mon.0) 810 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645942]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:31:59
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645942/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645942/

    "1712609741.41841 mon.smithi043 (mon.0) 948 : cluster [WRN] Health check
    failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

[7645943]  orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
-----------------------------------------------------------------
time:   00:39:19
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645943/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645943/

    "2024-04-08T20:46:27.056768+0000 mon.a (mon.0) 207 : cluster [WRN] Health
    check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

[7645944]  orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations}
-----------------------------------------------------------------
time:   00:16:17
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645944/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645944/

    "2024-04-08T20:48:47.511901+0000 mon.a (mon.0) 424 : cluster [WRN] Health
    check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in
    cluster log

[7645945]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
-----------------------------------------------------------------
time:   00:21:58
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645945/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645945/

    "2024-04-08T20:54:32.833595+0000 mon.smithi064 (mon.0) 864 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645946]  orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag}
-----------------------------------------------------------------
time:   00:13:48
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645946/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645946/

    "2024-04-08T20:47:22.733116+0000 mon.smithi114 (mon.0) 792 : cluster [WRN]
    Health check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645947]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
-----------------------------------------------------------------
time:   00:15:37
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645947/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645947/

    "2024-04-08T20:48:37.161350+0000 mon.smithi006 (mon.0) 816 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645948]  orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli}
-----------------------------------------------------------------
time:   00:14:21
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645948/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645948/

    "2024-04-08T20:48:18.252795+0000 mon.a (mon.0) 463 : cluster [WRN] Health
    check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645949]  orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_domain}
-----------------------------------------------------------------
time:   00:12:00
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645949/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645949/

    "2024-04-08T20:44:52.260216+0000 mon.a (mon.0) 244 : cluster [WRN] Health
    check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645950]  orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
-----------------------------------------------------------------
time:   00:15:13
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645950/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645950/

    "2024-04-08T20:47:10.216633+0000 mon.a (mon.0) 671 : cluster [WRN] Health
    check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645951]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
-----------------------------------------------------------------
time:   00:17:19
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645951/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645951/

    "2024-04-08T20:47:26.578308+0000 mon.smithi152 (mon.0) 868 : cluster [WRN]
    Health check failed: Failed to place 2 daemon(s)
    (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

[7645952]  orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
-----------------------------------------------------------------
time:   01:04:14
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645952/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645952/

    "2024-04-08T20:50:00.000188+0000 mon.a (mon.0) 704 : cluster [WRN] Health
    detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in
    cluster log

[7645953]  orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:44:44
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645953/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645953/

    "2024-04-08T21:10:00.000166+0000 mon.a (mon.0) 2593 : cluster [WRN] Health
    detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in
    cluster log

[7645955]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
-----------------------------------------------------------------
time:   00:23:57
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645955/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645955/

    "2024-04-08T20:54:07.425829+0000 mon.smithi044 (mon.0) 750 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645956]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   01:07:25
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645956/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645956/

    reached maximum tries (51) after waiting for 300 seconds

[7645957]  orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait}
-----------------------------------------------------------------
time:   00:13:43
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645957/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645957/

    "2024-04-08T20:49:44.595751+0000 mon.smithi186 (mon.0) 783 : cluster [WRN]
    Health check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645958]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final}
-----------------------------------------------------------------
time:   00:15:39
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645958/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645958/

    "2024-04-08T20:52:00.580645+0000 mon.smithi007 (mon.0) 838 : cluster [WRN]
    Health check failed: Failed to place 1 daemon(s)
    (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

[7645960]  orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:17:23
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645960/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645960/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9a8a3c9294a04cbaaedb8caf3bf2787d

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi102 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=fb6f9800392d73c10dc835c8e8e86c5884686270
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[7645961]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final}
-----------------------------------------------------------------
time:   00:14:17
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645961/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645961/

    "2024-04-08T21:00:32.955167+0000 mon.smithi150 (mon.0) 785 : cluster [WRN]
    Health check failed: Failed to place 1 daemon(s)
    (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

[7645962]  orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon}
-----------------------------------------------------------------
time:   00:36:44
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645962/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645962/

    "2024-04-08T21:06:40.341079+0000 mon.a (mon.0) 277 : cluster [WRN] Health
    detail: HEALTH_WARN 1/4 mons down, quorum a,e,c" in cluster log

[7645963]  orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_dom}
-----------------------------------------------------------------
time:   01:04:52
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645963/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645963/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fba69690b9ee4fc4b67cb5743a85575d

    reached maximum tries (301) after waiting for 300 seconds

[7645964]  orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final}
-----------------------------------------------------------------
time:   00:20:41
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645964/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645964/

    "2024-04-08T21:05:09.464957+0000 mon.smithi026 (mon.0) 576 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645965]  orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
-----------------------------------------------------------------
time:   00:31:44
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645965/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645965/

    "2024-04-08T21:07:08.682004+0000 mon.a (mon.0) 867 : cluster [WRN] Health
    check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log

[7645966]  orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:23:29
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645966/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645966/

    "2024-04-08T21:10:00.000185+0000 mon.a (mon.0) 1101 : cluster [WRN] Health
    detail: HEALTH_WARN 2 pool(s) do not have an application enabled" in
    cluster log

[7645968]  orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate}
-----------------------------------------------------------------
time:   00:21:39
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645968/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645968/

    "2024-04-08T21:08:43.705806+0000 mon.smithi038 (mon.0) 770 : cluster [WRN]
    Health check failed: 1 osds down (OSD_DOWN)" in cluster log

[7645969]  orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
-----------------------------------------------------------------
time:   00:31:57
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645969/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645969/

    "1712610663.5275354 mon.smithi092 (mon.0) 919 : cluster [WRN] Health check
    failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

[7645970]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final}
-----------------------------------------------------------------
time:   00:14:27
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645970/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645970/

    "2024-04-08T21:03:14.196465+0000 mon.smithi032 (mon.0) 769 : cluster [WRN]
    Health check failed: 1 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7645971]  orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic}
-----------------------------------------------------------------
time:   00:43:51
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645971/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645971/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e74b12ca3d24308983899241560f3e4

    Command failed on smithi012 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    4432bbac-f5ea-11ee-b649-cb9ed24678a4 -e
    sha1=fb6f9800392d73c10dc835c8e8e86c5884686270 -- bash -c \'ceph orch
    upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date |
    length == 7\'"\'"\'\''



Dead
=================================================================
[7645886]  orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645886/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645886/

    hit max job timeout

[7645894]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645894/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645894/

    hit max job timeout

[7645904]  orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645904/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645904/

    Error reimaging machines: Failed to power on smithi023

[7645905]  orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645905/
log:    http://qa-proxy.ceph.com/teuthology/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645905/

    Error reimaging machines: Failed to power on smithi154



Running
=================================================================
[7645959] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645959/



Pass
=================================================================
[7645891] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli}
time:   00:12:26
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645891/

[7645895] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final}
time:   00:18:43
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645895/

[7645897] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos}
time:   00:06:39
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645897/

[7645908] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final}
time:   00:14:25
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645908/

[7645914] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
time:   00:17:49
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645914/

[7645919] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python}
time:   00:23:17
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645919/

[7645921] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption}
time:   00:12:27
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645921/

[7645925] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final}
time:   00:13:41
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645925/

[7645936] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli}
time:   00:12:38
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645936/

[7645938] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final}
time:   00:14:29
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645938/

[7645939] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final}
time:   00:12:23
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645939/

[7645954] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key}
time:   00:12:46
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645954/

[7645967] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos}
time:   00:06:31
info:   https://pulpito.ceph.com/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645967/