2024-03-26T20:32:17.186 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-26T20:32:17.187 INFO:teuthology.results:subset: '1/8' 2024-03-26T20:32:17.187 INFO:teuthology.results:seed: '832' 2024-03-26T21:36:18.583 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-26T21:36:18.605 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-27T08:44:15.058 INFO:teuthology.results:Tests finished! gathering results... 2024-03-27T08:44:23.619 INFO:scrape:Found 107 jobs 2024-03-27T08:44:35.784 WARNING:scrape:Saw end of BT but not start 2024-03-27T08:44:35.784 WARNING:scrape:Saw end of BT but not start 2024-03-27T08:44:35.784 WARNING:scrape:Saw end of BT but not start 2024-03-27T08:44:47.367 INFO:scrape:Found 7 distinct failure reasons 2024-03-27T08:44:47.368 INFO:scrape:Failure: "2024-03-26T21:49:04.680646+0000 mon.smithi053 (mon.0) 804 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2024-03-27T08:44:47.368 INFO:scrape:87 jobs 2024-03-27T08:44:47.369 INFO:scrape:suites intersection: [] 2024-03-27T08:44:47.369 INFO:scrape:suites union: ['0-nvme-loop', '1-bootstrap/17.2.0', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '2-nfs', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/iscsi', '2-services/jaeger', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-inline/no', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-final}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'base/install', 'centos_9.stream', 'cluster/{fixed-3', 'conf/{client', 'conf/{disable-pool-app}', 'fail_fs/no', 'fixed-2', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream', 'orch:cephadm/nfs/{cluster/{1-node}', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04', 'orch:cephadm/osds/{0-distro/centos_9.stream', 'orch:cephadm/osds/{0-distro/centos_9.stream_runc', 'orch:cephadm/osds/{0-distro/ubuntu_22.04', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smb/{0-distro/centos_9.stream', 'orch:cephadm/smb/{0-distro/centos_9.stream_runc', 'orch:cephadm/smb/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc}', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/centos_9.stream', 'orch:cephadm/smoke/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/ubuntu_22.04', 'orch:cephadm/thrash/{0-distro/centos_9.stream', 'orch:cephadm/thrash/{0-distro/centos_9.stream_runc', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_9.stream', 'orch:cephadm/with-work/{0-distro/centos_9.stream_runc', 'orch:cephadm/with-work/{0-distro/ubuntu_22.04', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'roles', 'root}', 'start', 'start}', 'supported-random-distros$/{ubuntu_latest}', 'syntax}', 'task/test_cephadm_timeout}', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_iscsi_container/{centos_9.stream', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/deploy_smb_basic}', 'tasks/deploy_smb_domain}', 'tasks/nfs}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'tasks/{0-from/quincy', 'test_iscsi_container}}', 'workloads/cephadm_iscsi}'] 2024-03-27T08:44:47.369 INFO:scrape: 2024-03-27T08:44:47.369 INFO:scrape:Dead: b'2024-03-27T08:41:40.348 DEBUG:teuthology.exit:Finished running handlers' 2024-03-27T08:44:47.369 INFO:scrape:1 jobs: ['7624085'] 2024-03-27T08:44:47.369 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream'] 2024-03-27T08:44:47.369 INFO:scrape: 2024-03-27T08:44:47.370 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi194 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-03-27T08:44:47.370 INFO:scrape:1 jobs: ['7624127'] 2024-03-27T08:44:47.370 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] 2024-03-27T08:44:47.370 INFO:scrape: 2024-03-27T08:44:47.370 INFO:scrape:Failure: "1711486800.000299 mon.smithi026 (mon.0) 659 : cluster [ERR] fs cephfs is offline because no MDS is active for it." in cluster log 2024-03-27T08:44:47.370 INFO:scrape:1 jobs: ['7624078'] 2024-03-27T08:44:47.370 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-27T08:44:47.370 INFO:scrape: 2024-03-27T08:44:47.370 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-27T08:44:47.370 INFO:scrape:2 jobs: ['7624149', '7624099'] 2024-03-27T08:44:47.370 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}'] 2024-03-27T08:44:47.371 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}'] 2024-03-27T08:44:47.371 INFO:scrape: 2024-03-27T08:44:47.371 INFO:scrape:Failure: Command failed on smithi196 with status 5: 'sudo systemctl stop ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196' 2024-03-27T08:44:47.371 INFO:scrape:1 jobs: ['7624176'] 2024-03-27T08:44:47.371 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}'] 2024-03-27T08:44:47.371 INFO:scrape: 2024-03-27T08:44:47.371 INFO:scrape:Failure: timeout expired in wait_until_healthy 2024-03-27T08:44:47.371 INFO:scrape:1 jobs: ['7624124'] 2024-03-27T08:44:47.371 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}'] 2024-03-27T08:44:47.371 INFO:scrape: 2024-03-27T08:44:47.375 INFO:teuthology.results:Sending results to gabrioux@ibm.com: Test Run: gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/ failed: 93 dead: 1 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [7624076] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:22:36 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624076/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624076/ "2024-03-26T21:00:00.000184+0000 mon.a (mon.0) 1174 : cluster [WRN] Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled" in cluster log [7624077] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:16:00 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624077/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624077/ "2024-03-26T20:52:16.917505+0000 mon.a (mon.0) 394 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7624078] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:00 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624078/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624078/ "1711486800.000299 mon.smithi026 (mon.0) 659 : cluster [ERR] fs cephfs is offline because no MDS is active for it." in cluster log [7624079] orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:34:46 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624079/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624079/ "2024-03-26T20:53:30.336493+0000 mon.smithi063 (mon.0) 503 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7624080] orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:48:31 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624080/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624080/ "2024-03-26T21:00:13.555405+0000 mon.a (mon.0) 325 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log [7624081] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:13:20 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624081/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624081/ "2024-03-26T20:53:25.605270+0000 mon.a (mon.0) 459 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624083] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:30:35 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624083/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624083/ "2024-03-26T20:54:57.939426+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7624084] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} ----------------------------------------------------------------- time: 00:13:12 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624084/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624084/ "2024-03-26T20:55:46.538892+0000 mon.a (mon.0) 244 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624087] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:21 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624087/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624087/ "2024-03-26T20:53:48.687519+0000 mon.a (mon.0) 469 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624088] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:58:02 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624088/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624088/ "2024-03-26T21:03:24.089547+0000 mon.a (mon.0) 865 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log [7624089] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:47:31 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624089/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624089/ "2024-03-26T20:55:56.782307+0000 mon.a (mon.0) 478 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7624090] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:23:49 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624090/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624090/ "2024-03-26T21:00:22.540078+0000 mon.a (mon.0) 661 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624091] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:13:47 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624091/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624091/ "2024-03-26T20:58:35.663940+0000 mon.a (mon.0) 303 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624092] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:17:45 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624092/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624092/ "2024-03-26T21:03:12.860147+0000 mon.smithi096 (mon.0) 786 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624093] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:37 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624093/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624093/ "2024-03-26T21:08:16.700989+0000 mon.smithi042 (mon.0) 770 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624094] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:20:52 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624094/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624094/ "2024-03-26T21:07:31.377870+0000 mon.smithi131 (mon.0) 825 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7624095] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:25:59 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624095/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624095/ "2024-03-26T21:04:19.886017+0000 mon.a (mon.0) 383 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7624096] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:32:17 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624096/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624096/ "2024-03-26T21:07:28.348778+0000 mon.a (mon.0) 830 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log [7624097] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 01:00:30 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624097/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624097/ "2024-03-26T21:20:00.000154+0000 mon.a (mon.0) 1372 : cluster [WRN] Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 5 pool(s) do not have an application enabled" in cluster log [7624098] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:25:22 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624098/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624098/ "2024-03-26T21:09:31.906419+0000 mon.a (mon.0) 439 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7624099] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:08:45 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624099/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624099/ reached maximum tries (51) after waiting for 300 seconds [7624100] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:15:31 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624100/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624100/ "2024-03-26T21:06:49.893541+0000 mon.smithi018 (mon.0) 799 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7624101] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:28:59 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624101/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624101/ "2024-03-26T21:15:20.119332+0000 mon.a (mon.0) 592 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624102] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} ----------------------------------------------------------------- time: 00:11:08 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624102/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624102/ "2024-03-26T21:05:39.098772+0000 mon.a (mon.0) 278 : cluster [WRN] Health check failed: Degraded data redundancy: 22/66 objects degraded (33.333%), 10 pgs degraded (PG_DEGRADED)" in cluster log [7624103] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:14:20 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624103/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624103/ "2024-03-26T21:10:10.517137+0000 mon.smithi183 (mon.0) 785 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624105] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:15:45 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624105/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624105/ "2024-03-26T21:14:45.500780+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7624106] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:43 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624106/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624106/ "2024-03-26T21:12:34.337317+0000 mon.smithi089 (mon.0) 784 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624107] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:21:57 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624107/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624107/ "2024-03-26T21:20:03.108250+0000 mon.smithi029 (mon.0) 785 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624108] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:50:18 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624108/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624108/ "2024-03-26T21:26:47.611422+0000 mon.a (mon.0) 947 : cluster [ERR] Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL)" in cluster log [7624109] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:38:48 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624109/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624109/ "1711488570.7345958 mon.smithi103 (mon.0) 955 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7624110] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:17:46 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624110/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624110/ "2024-03-26T21:13:28.574793+0000 mon.smithi102 (mon.0) 800 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624111] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:54 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624111/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624111/ "2024-03-26T21:09:15.194789+0000 mon.a (mon.0) 265 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7624112] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:41:08 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624112/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624112/ "2024-03-26T21:14:00.254269+0000 mon.a (mon.0) 432 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7624113] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:26:50 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624113/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624113/ "2024-03-26T21:18:23.650191+0000 mon.a (mon.0) 663 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624114] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:16:06 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624114/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624114/ "2024-03-26T21:16:15.230225+0000 mon.a (mon.0) 266 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7624115] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:18:14 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624115/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624115/ "2024-03-26T21:17:15.213512+0000 mon.smithi045 (mon.0) 797 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624118] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} ----------------------------------------------------------------- time: 00:19:04 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624118/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624118/ "2024-03-26T21:21:06.091537+0000 mon.a (mon.0) 289 : cluster [WRN] Health check failed: Degraded data redundancy: 7/21 objects degraded (33.333%), 3 pgs degraded (PG_DEGRADED)" in cluster log [7624119] orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:15:50 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624119/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624119/ "2024-03-26T21:14:00.881095+0000 mon.a (mon.0) 414 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7624121] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:54:22 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624121/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624121/ "2024-03-26T21:33:27.738501+0000 mon.a (mon.0) 928 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log [7624122] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:53:01 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624122/ "2024-03-26T21:30:00.000178+0000 mon.a (mon.0) 2476 : cluster [WRN] Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled" in cluster log [7624124] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:23:05 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624124/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624124/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48f2bdcd94934223b1c1ead80438692d timeout expired in wait_until_healthy [7624126] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:32:31 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624126/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624126/ "2024-03-26T21:19:41.761510+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7624127] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:17:27 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624127/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624127/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77ff4a7b65814a42b807c417964d9197 Command failed (workunit test cephadm/test_cephadm.sh) on smithi194 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7624128] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:14:32 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624128/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624128/ "2024-03-26T21:26:45.936072+0000 mon.smithi145 (mon.0) 784 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624129] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 01:09:03 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129/ "2024-03-26T21:30:30.725236+0000 mon.a (mon.0) 835 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624130] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:21:49 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624130/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624130/ "2024-03-26T21:30:43.564924+0000 mon.smithi022 (mon.0) 783 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7624131] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:21:54 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624131/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624131/ "2024-03-26T21:31:17.563267+0000 mon.smithi136 (mon.0) 768 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624132] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:25:35 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624132/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624132/ "2024-03-26T21:30:05.309045+0000 mon.a (mon.0) 661 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624134] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:34 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624134/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624134/ "1711488762.4052248 mon.smithi012 (mon.0) 935 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7624135] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:14:07 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624135/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624135/ "2024-03-26T21:26:27.001637+0000 mon.a (mon.0) 211 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log [7624137] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} ----------------------------------------------------------------- time: 00:11:28 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624137/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624137/ "2024-03-26T21:23:09.178042+0000 mon.a (mon.0) 280 : cluster [WRN] Health check failed: Degraded data redundancy: 21/63 objects degraded (33.333%), 10 pgs degraded (PG_DEGRADED)" in cluster log [7624139] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:13:12 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624139/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624139/ "2024-03-26T21:25:21.895727+0000 mon.smithi184 (mon.0) 242 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624140] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:18:34 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624140/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624140/ "2024-03-26T21:29:14.824657+0000 mon.a (mon.0) 393 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7624141] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:53:10 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624141/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624141/ "2024-03-26T21:26:13.266415+0000 mon.a (mon.0) 195 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7624142] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:15:35 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624142/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624142/ "2024-03-26T21:39:51.144356+0000 mon.smithi031 (mon.0) 572 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624143] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:15:21 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624143/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624143/ "2024-03-26T21:40:12.226846+0000 mon.a (mon.0) 664 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624144] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:32:27 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624144/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624144/ "2024-03-26T21:44:29.334225+0000 mon.a (mon.0) 645 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7624145] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:53:02 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624145/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624145/ "2024-03-26T21:48:27.232463+0000 mon.a (mon.0) 960 : cluster [WRN] Health check failed: 17 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7624146] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:14:03 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624146/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624146/ "2024-03-26T21:40:42.930071+0000 mon.a (mon.0) 305 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7624147] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:22:13 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624147/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624147/ "2024-03-26T21:49:04.680646+0000 mon.smithi053 (mon.0) 804 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624148] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:14:17 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624148/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624148/ "2024-03-26T21:43:57.795378+0000 mon.smithi111 (mon.0) 798 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624149] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:40:02 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624149/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624149/ reached maximum tries (51) after waiting for 300 seconds [7624150] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:15:29 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624150/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624150/ "2024-03-26T21:40:31.128482+0000 mon.smithi007 (mon.0) 794 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624151] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:32:58 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624151/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624151/ "2024-03-26T21:43:50.572555+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7624152] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:23:33 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624152/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624152/ "2024-03-26T21:52:23.851939+0000 mon.a (mon.0) 641 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624153] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:17:20 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624153/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624153/ "2024-03-26T21:44:29.421261+0000 mon.a (mon.0) 465 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624154] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} ----------------------------------------------------------------- time: 00:13:22 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624154/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624154/ "2024-03-26T21:42:38.782478+0000 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624155] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:17:28 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624155/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624155/ "2024-03-26T21:43:36.041210+0000 mon.smithi089 (mon.0) 835 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7624156] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:48:38 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624156/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624156/ "2024-03-26T21:55:05.805962+0000 mon.a (mon.0) 918 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log [7624157] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:26:10 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624157/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624157/ "2024-03-26T21:46:43.154891+0000 mon.a (mon.0) 667 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624158] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} ----------------------------------------------------------------- time: 00:13:53 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624158/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624158/ "2024-03-26T21:39:44.645346+0000 mon.a (mon.0) 186 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7624159] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:25:10 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624159/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624159/ "2024-03-26T21:49:10.654887+0000 mon.smithi112 (mon.0) 774 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624160] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:14:12 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624160/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624160/ "2024-03-26T21:43:34.380760+0000 mon.smithi131 (mon.0) 799 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624161] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:40 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624161/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624161/ "1711489825.3655252 mon.smithi148 (mon.0) 913 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7624162] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:15:32 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624162/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624162/ "2024-03-26T21:41:57.797436+0000 mon.smithi049 (mon.0) 576 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624163] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:11:57 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624163/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624163/ "2024-03-26T21:40:07.266504+0000 mon.a (mon.0) 463 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624164] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:37:49 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624164/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624164/ "2024-03-26T21:43:34.697192+0000 mon.a (mon.0) 474 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7624165] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:25:34 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624165/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624165/ "2024-03-26T21:46:46.106281+0000 mon.a (mon.0) 519 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7624166] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:35:18 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624166/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624166/ "2024-03-26T21:54:33.074133+0000 mon.a (mon.0) 833 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log [7624167] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:53:00 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624167/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624167/ "2024-03-26T21:56:44.324087+0000 mon.a (mon.0) 1119 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7624168] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:20:33 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624168/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624168/ "2024-03-26T21:58:10.528752+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [7624169] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:15:21 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624169/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624169/ "2024-03-26T21:57:51.164031+0000 mon.smithi098 (mon.0) 785 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7624170] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:37:38 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624170/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624170/ "2024-03-26T22:02:00.555410+0000 mon.a (mon.0) 244 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,e" in cluster log [7624171] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} ----------------------------------------------------------------- time: 00:18:28 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624171/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624171/ "2024-03-26T22:02:14.228107+0000 mon.a (mon.0) 247 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624172] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:22:14 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624172/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624172/ "2024-03-26T22:01:26.885679+0000 mon.smithi121 (mon.0) 576 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624173] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:34:03 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624173/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624173/ "2024-03-26T21:58:47.110603+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7624174] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:20:16 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624174/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624174/ "2024-03-26T22:00:47.484333+0000 mon.a (mon.0) 443 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7624175] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:21:40 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624175/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624175/ "2024-03-26T22:03:47.403535+0000 mon.smithi082 (mon.0) 765 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7624176] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:24:41 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=982c56716ff144fd8ff04f2fd2049197 Command failed on smithi196 with status 5: 'sudo systemctl stop ceph-2f9b9224-ebbb-11ee-95d0-87774f69a715@mon.smithi196' [7624177] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:14:16 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624177/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624177/ "2024-03-26T21:57:33.078598+0000 mon.smithi045 (mon.0) 782 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624178] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} ----------------------------------------------------------------- time: 00:59:24 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624178/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624178/ "2024-03-26T22:02:14.249622+0000 mon.a (mon.0) 882 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log [7624179] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:18:31 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624179/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624179/ "2024-03-26T21:57:21.024273+0000 mon.smithi029 (mon.0) 807 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7624180] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} ----------------------------------------------------------------- time: 00:24:58 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624180/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624180/ "2024-03-26T22:00:00.942201+0000 mon.a (mon.0) 580 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log Dead ================================================================= [7624085] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624085/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624085/ hit max job timeout Pass ================================================================= [7624082] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} time: 00:14:11 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624082/ [7624086] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} time: 00:18:10 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624086/ [7624104] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:22:08 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624104/ [7624116] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} time: 00:12:35 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624116/ [7624117] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} time: 00:13:32 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624117/ [7624120] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:19:05 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624120/ [7624123] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} time: 00:11:47 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624123/ [7624125] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:13:47 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624125/ [7624133] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} time: 00:06:37 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624133/ [7624136] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} time: 00:16:11 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624136/ [7624138] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:16:57 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624138/ [7624181] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:17:27 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624181/ [7624182] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:21:44 info: https://pulpito.ceph.com/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624182/