2024-03-26T01:51:17.707 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-26T01:51:17.707 INFO:teuthology.results:subset: '1/16' 2024-03-26T01:51:17.707 INFO:teuthology.results:seed: '9192' 2024-03-26T03:20:21.184 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-26T03:20:21.218 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-26T15:09:17.038 INFO:teuthology.results:Tests finished! gathering results... 2024-03-26T15:09:24.459 INFO:scrape:Found 98 jobs 2024-03-26T15:09:43.782 INFO:scrape:Found 4 distinct failure reasons 2024-03-26T15:09:43.783 INFO:scrape:Failure: "2024-03-26T03:21:13.143447+0000 mon.a (mon.0) 639 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log 2024-03-26T15:09:43.783 INFO:scrape:74 jobs 2024-03-26T15:09:43.783 INFO:scrape:suites intersection: [] 2024-03-26T15:09:43.784 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '2-nfs', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-inline/no', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'base/install', 'centos_9.stream', 'cluster/{fixed-3', 'conf/{client', 'conf/{disable-pool-app}', 'fail_fs/no', 'fixed-2', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/nfs/{cluster/{1-node}', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04', 'orch:cephadm/osds/{0-distro/centos_9.stream', 'orch:cephadm/osds/{0-distro/ubuntu_22.04', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smb/{0-distro/centos_9.stream', 'orch:cephadm/smb/{0-distro/centos_9.stream_runc', 'orch:cephadm/smb/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc}', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/centos_9.stream', 'orch:cephadm/smoke/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/ubuntu_22.04', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_9.stream', 'orch:cephadm/with-work/{0-distro/centos_9.stream_runc', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'roles', 'root}', 'start', 'start}', 'supported-random-distros$/{centos_latest}', 'syntax}', 'task/test_cephadm_timeout}', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_iscsi_container/{centos_9.stream', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/deploy_smb_basic}', 'tasks/deploy_smb_domain}', 'tasks/nfs}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'tasks/{0-from/quincy', 'test_iscsi_container}}', 'workloads/cephadm_iscsi}'] 2024-03-26T15:09:43.784 INFO:scrape: 2024-03-26T15:09:43.784 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-26T15:09:43.784 INFO:scrape:4 jobs: ['7622377', '7622448', '7622401', '7622423'] 2024-03-26T15:09:43.784 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-03-26T15:09:43.784 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-03-26T15:09:43.784 INFO:scrape: 2024-03-26T15:09:43.785 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-03-26T15:09:43.785 INFO:scrape:1 jobs: ['7622403'] 2024-03-26T15:09:43.785 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] 2024-03-26T15:09:43.785 INFO:scrape: 2024-03-26T15:09:43.785 INFO:scrape:Dead: b'2024-03-26T14:55:54.827 DEBUG:teuthology.exit:Finished running handlers' 2024-03-26T15:09:43.785 INFO:scrape:2 jobs: ['7622431', '7622433'] 2024-03-26T15:09:43.785 INFO:scrape:suites intersection: [] 2024-03-26T15:09:43.785 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', 'agent/on', 'mon_election/connectivity', 'orch:cephadm/osds/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'task/test_iscsi_container/{centos_9.stream', 'test_iscsi_container}}'] 2024-03-26T15:09:43.785 INFO:scrape: 2024-03-26T15:09:43.789 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/ failed: 79 dead: 2 running: 0 waiting: 0 queued: 0 passed: 17 Fail ================================================================= [7622356] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:21:33 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622356/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622356/ "2024-03-26T02:18:47.148265+0000 mon.smithi118 (mon.0) 780 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622357] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:17:55 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622357/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622357/ "2024-03-26T02:14:48.048351+0000 mon.a (mon.0) 380 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622358] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:37 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622358/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622358/ "1711419753.0212104 mon.smithi063 (mon.0) 905 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622359] orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:34:44 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622359/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622359/ "2024-03-26T02:22:21.808791+0000 mon.smithi136 (mon.0) 879 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622360] orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:35:09 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622360/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622360/ "2024-03-26T02:17:06.454878+0000 mon.a (mon.0) 613 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7622361] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:13:38 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622361/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622361/ "2024-03-26T02:15:53.475511+0000 mon.a (mon.0) 269 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622363] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:26:22 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622363/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622363/ "2024-03-26T02:12:44.763911+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7622364] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} ----------------------------------------------------------------- time: 00:13:38 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622364/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622364/ "2024-03-26T02:18:31.291118+0000 mon.a (mon.0) 289 : cluster [WRN] Health check failed: Degraded data redundancy: 22/66 objects degraded (33.333%), 10 pgs degraded (PG_DEGRADED)" in cluster log [7622365] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:17:58 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622365/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622365/ "2024-03-26T02:17:52.860672+0000 mon.smithi029 (mon.0) 742 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7622367] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:12:15 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622367/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622367/ "2024-03-26T02:24:04.465211+0000 mon.a (mon.0) 443 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622368] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:48:26 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368/ "2024-03-26T02:31:54.373419+0000 mon.a (mon.0) 968 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622369] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:36:35 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622369/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622369/ "2024-03-26T02:24:33.640311+0000 mon.a (mon.0) 553 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7622370] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:22:57 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622370/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622370/ "2024-03-26T02:30:00.000194+0000 mon.a (mon.0) 1104 : cluster [WRN] Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled" in cluster log [7622371] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:15:11 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622371/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622371/ "2024-03-26T02:23:44.949061+0000 mon.a (mon.0) 303 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622373] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:41 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622373/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622373/ "2024-03-26T02:28:54.909915+0000 mon.smithi102 (mon.0) 775 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7622374] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:22:21 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622374/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622374/ "2024-03-26T02:31:06.239301+0000 mon.smithi055 (mon.0) 783 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622375] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:26:08 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622375/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622375/ "2024-03-26T02:25:19.334478+0000 mon.a (mon.0) 458 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7622376] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:25:17 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622376/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622376/ "2024-03-26T02:26:57.882196+0000 mon.a (mon.0) 451 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7622377] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:06:34 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622377/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622377/ reached maximum tries (51) after waiting for 300 seconds [7622378] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:15:07 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622378/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622378/ "2024-03-26T02:30:15.733313+0000 mon.smithi053 (mon.0) 582 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622379] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:25:04 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622379/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622379/ "2024-03-26T02:45:29.777294+0000 mon.a (mon.0) 1024 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log [7622380] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} ----------------------------------------------------------------- time: 00:11:41 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622380/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622380/ "2024-03-26T02:35:55.424235+0000 mon.a (mon.0) 245 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622381] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:17:28 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622381/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622381/ "2024-03-26T02:38:56.834942+0000 mon.smithi044 (mon.0) 303 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7622382] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} ----------------------------------------------------------------- time: 01:04:15 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622382/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622382/ "2024-03-26T02:54:32.881453+0000 mon.a (mon.0) 1156 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL)" in cluster log [7622383] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:39:10 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622383/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622383/ "2024-03-26T02:47:24.336858+0000 mon.a (mon.0) 656 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7622385] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:24 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622385/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622385/ "2024-03-26T02:38:19.705736+0000 mon.smithi150 (mon.0) 781 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7622386] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:20:03 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622386/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622386/ "2024-03-26T02:43:48.615395+0000 mon.smithi080 (mon.0) 749 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622387] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:34:51 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622387/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622387/ "1711421342.7324953 mon.smithi057 (mon.0) 903 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622388] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:18:02 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622388/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622388/ "2024-03-26T02:39:51.686531+0000 mon.smithi016 (mon.0) 807 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622390] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:43:43 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622390/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622390/ "2024-03-26T02:41:08.515803+0000 mon.a (mon.0) 433 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7622391] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:16:03 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622391/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622391/ "2024-03-26T02:47:14.970452+0000 mon.a (mon.0) 374 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622392] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:15:55 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622392/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622392/ "2024-03-26T02:55:15.415165+0000 mon.smithi001 (mon.0) 826 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622395] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} ----------------------------------------------------------------- time: 00:18:59 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622395/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622395/ "2024-03-26T03:00:13.399789+0000 mon.a (mon.0) 290 : cluster [WRN] Replacing daemon mds.cephfs.smithi031.okjpbc as rank 0 with standby daemon mds.cephfs.smithi031.nifdkr" in cluster log [7622396] orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:15:19 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622396/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622396/ "2024-03-26T02:53:01.265885+0000 mon.a (mon.0) 588 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622398] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:54:41 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398/ "2024-03-26T03:04:14.387440+0000 mon.a (mon.0) 1051 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY)" in cluster log [7622399] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 01:03:35 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622399/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622399/ "2024-03-26T03:12:07.519796+0000 mon.a (mon.0) 1184 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7622401] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:07:09 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622401/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622401/ reached maximum tries (51) after waiting for 300 seconds [7622402] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} ----------------------------------------------------------------- time: 00:14:01 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622402/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622402/ "2024-03-26T02:54:54.200253+0000 mon.smithi113 (mon.0) 571 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622403] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:17:16 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622403/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622403/ sentry: https://sentry.ceph.com/organizations/ceph/?query=620442aba6304d50807d08e8c8a502d5 Command failed (workunit test cephadm/test_cephadm.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7622404] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:15:09 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622404/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622404/ "2024-03-26T02:55:03.454796+0000 mon.smithi029 (mon.0) 781 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622405] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:20:57 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622405/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622405/ "2024-03-26T03:01:21.795463+0000 mon.smithi102 (mon.0) 809 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622406] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:19:45 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622406/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622406/ "2024-03-26T02:56:09.941629+0000 mon.smithi047 (mon.0) 676 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7622407] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:30:39 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622407/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622407/ "2024-03-26T03:05:43.526091+0000 mon.a (mon.0) 1555 : cluster [ERR] Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL)" in cluster log [7622410] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:34:04 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622410/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622410/ "1711422356.7341175 mon.smithi006 (mon.0) 937 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622411] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:14:33 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622411/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622411/ "2024-03-26T03:04:16.895658+0000 mon.a (mon.0) 211 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log [7622413] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} ----------------------------------------------------------------- time: 00:11:05 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622413/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622413/ "2024-03-26T03:06:17.494462+0000 mon.a (mon.0) 280 : cluster [WRN] Health check failed: Degraded data redundancy: 21/63 objects degraded (33.333%), 10 pgs degraded (PG_DEGRADED)" in cluster log [7622415] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:13:01 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622415/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622415/ "2024-03-26T03:10:04.433939+0000 mon.smithi016 (mon.0) 444 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622416] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:12:12 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622416/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622416/ "2024-03-26T03:08:39.604331+0000 mon.a (mon.0) 465 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622417] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:51:06 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622417/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622417/ "2024-03-26T03:05:28.007053+0000 mon.a (mon.0) 356 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7622418] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:15:17 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622418/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622418/ "2024-03-26T03:07:12.365442+0000 mon.smithi003 (mon.0) 588 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622419] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:18:57 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622419/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622419/ "2024-03-26T03:09:03.586293+0000 mon.a (mon.0) 658 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622420] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:18:36 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622420/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622420/ "2024-03-26T03:15:43.327378+0000 mon.a (mon.0) 52 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622421] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:21:55 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622421/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622421/ "2024-03-26T03:16:37.500334+0000 mon.smithi046 (mon.0) 794 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622422] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:53 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622422/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622422/ "2024-03-26T03:11:07.081175+0000 mon.smithi073 (mon.0) 793 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7622423] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:40:08 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622423/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622423/ reached maximum tries (51) after waiting for 300 seconds [7622424] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:17:51 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622424/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622424/ "2024-03-26T03:13:37.937447+0000 mon.smithi088 (mon.0) 794 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622425] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:49:01 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622425/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622425/ "2024-03-26T03:29:34.546566+0000 mon.a (mon.0) 1316 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7622427] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:23:01 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622427/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622427/ "2024-03-26T03:21:13.143447+0000 mon.a (mon.0) 639 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7622428] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:13:09 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622428/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622428/ "2024-03-26T03:10:12.699745+0000 mon.a (mon.0) 464 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7622429] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} ----------------------------------------------------------------- time: 00:15:43 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622429/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622429/ "2024-03-26T03:12:52.018644+0000 mon.a (mon.0) 277 : cluster [WRN] Health check failed: Degraded data redundancy: 22/66 objects degraded (33.333%), 10 pgs degraded (PG_DEGRADED)" in cluster log [7622430] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:22:53 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622430/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622430/ "2024-03-26T03:16:52.024935+0000 mon.smithi080 (mon.0) 880 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622432] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:23:15 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622432/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622432/ "2024-03-26T03:23:05.433132+0000 mon.smithi043 (mon.0) 820 : cluster [WRN] Replacing daemon mds.foofs.smithi118.vsrsbe as rank 0 with standby daemon mds.foofs.smithi043.oqleun" in cluster log [7622434] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:42 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622434/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622434/ "1711423825.5449877 mon.smithi067 (mon.0) 915 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622435] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:15:48 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622435/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622435/ "2024-03-26T03:24:21.003115+0000 mon.smithi119 (mon.0) 791 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622436] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:35 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622436/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622436/ "2024-03-26T03:22:40.746736+0000 mon.a (mon.0) 429 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622437] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:38:01 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622437/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622437/ "2024-03-26T03:46:24.067027+0000 mon.a (mon.0) 1012 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7622438] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:26:21 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622438/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622438/ "2024-03-26T03:35:26.332589+0000 mon.a (mon.0) 386 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7622439] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 01:06:58 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622439/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622439/ "2024-03-26T03:49:54.631194+0000 mon.a (mon.0) 1286 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL)" in cluster log [7622440] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 01:01:06 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622440/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622440/ "2024-03-26T03:50:00.000143+0000 mon.a (mon.0) 1382 : cluster [WRN] Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 5 pool(s) do not have an application enabled" in cluster log [7622442] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:14:56 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622442/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622442/ "2024-03-26T03:35:57.879535+0000 mon.smithi045 (mon.0) 789 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7622443] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:37:40 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622443/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622443/ "2024-03-26T03:51:43.174294+0000 mon.a (mon.0) 833 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7622444] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} ----------------------------------------------------------------- time: 00:18:15 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622444/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622444/ "2024-03-26T03:43:19.175864+0000 mon.a (mon.0) 291 : cluster [WRN] Replacing daemon mds.cephfs.smithi037.blrdoq as rank 0 with standby daemon mds.cephfs.smithi037.abwhjt" in cluster log [7622446] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:20:03 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622446/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622446/ "2024-03-26T03:45:10.096198+0000 mon.a (mon.0) 451 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7622447] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:22:26 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622447/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622447/ "2024-03-26T03:45:25.209989+0000 mon.smithi044 (mon.0) 564 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622448] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:38:10 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622448/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622448/ reached maximum tries (51) after waiting for 300 seconds [7622449] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:14:38 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622449/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622449/ "2024-03-26T03:40:55.435988+0000 mon.smithi003 (mon.0) 767 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622450] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:18:27 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622450/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622450/ "2024-03-26T03:41:42.502440+0000 mon.smithi121 (mon.0) 802 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7622451] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 01:16:45 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451/ "2024-03-26T04:02:29.491553+0000 mon.a (mon.0) 1430 : cluster [ERR] Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL)" in cluster log [7622452] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:23:10 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622452/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622452/ "2024-03-26T03:50:00.000078+0000 mon.a (mon.0) 1374 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log Dead ================================================================= [7622431] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622431/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622431/ hit max job timeout [7622433] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622433/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622433/ hit max job timeout Pass ================================================================= [7622362] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli} time: 00:12:13 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622362/ [7622366] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} time: 00:11:23 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622366/ [7622372] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:16:38 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622372/ [7622384] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} time: 00:13:59 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622384/ [7622389] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:24:53 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622389/ [7622393] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} time: 00:12:49 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622393/ [7622394] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} time: 00:13:13 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622394/ [7622397] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:19:25 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622397/ [7622400] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} time: 00:12:08 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622400/ [7622408] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:23:29 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622408/ [7622409] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} time: 00:06:33 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622409/ [7622412] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} time: 00:14:50 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622412/ [7622414] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:14:31 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622414/ [7622426] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys} time: 00:23:37 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622426/ [7622441] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:18:51 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622441/ [7622445] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:20:16 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622445/ [7622453] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:17:15 info: https://pulpito.ceph.com/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622453/