2024-04-09T04:09:59.861 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-09T04:09:59.861 INFO:teuthology.results:subset: '1/16' 2024-04-09T04:09:59.861 INFO:teuthology.results:seed: '8760' 2024-04-09T05:01:30.098 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-09T05:01:30.126 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-09T05:52:34.473 INFO:teuthology.results:Tests finished! gathering results... 2024-04-09T05:52:38.733 INFO:scrape:Found 83 jobs 2024-04-09T05:52:44.944 WARNING:scrape:Saw end of BT but not start 2024-04-09T05:52:44.944 WARNING:scrape:Saw end of BT but not start 2024-04-09T05:52:44.945 WARNING:scrape:Saw end of BT but not start 2024-04-09T05:52:46.362 INFO:scrape:Found 3 distinct failure reasons 2024-04-09T05:52:46.362 INFO:scrape:Failure: "2024-04-09T05:14:48.028214+0000 mon.a (mon.0) 408 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log 2024-04-09T05:52:46.362 INFO:scrape:40 jobs 2024-04-09T05:52:46.363 INFO:scrape:suites intersection: [] 2024-04-09T05:52:46.363 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/basic', '2-services/iscsi', '2-services/jaeger', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '2-thrash', '3-final}', '3-tasks/rados_api_tests', '3-upgrade/simple', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'conf/{client', 'conf/{disable-pool-app}', 'fixed-2', 'mds', 'mgr', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async-v1only', 'openstack}', 'orch:cephadm/nfs/{cluster/{1-node}', 'orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream', 'orch:cephadm/osds/{0-distro/centos_9.stream', 'orch:cephadm/osds/{0-distro/ubuntu_22.04', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream', 'orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream}', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04}', 'orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc', 'orch:cephadm/smoke/{0-distro/centos_9.stream', 'orch:cephadm/smoke/{0-distro/ubuntu_22.04', 'orch:cephadm/thrash/{0-distro/ubuntu_22.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04', 'orch:cephadm/workunits/{0-distro/centos_9.stream', 'orch:cephadm/workunits/{0-distro/centos_9.stream_runc', 'orch:cephadm/workunits/{0-distro/ubuntu_22.04', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'root}', 'start}', 'supported-random-distros$/{centos_latest}', 'task/test_cephadm_timeout}', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/nfs}', 'workloads/cephadm_iscsi}'] 2024-04-09T05:52:46.363 INFO:scrape: 2024-04-09T05:52:46.363 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-04-09T05:52:46.363 INFO:scrape:2 jobs: ['7648059', '7648021'] 2024-04-09T05:52:46.363 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/no', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-04-09T05:52:46.363 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-04-09T05:52:46.364 INFO:scrape: 2024-04-09T05:52:46.364 INFO:scrape:Failure: "2024-04-09T05:20:00.000126+0000 mon.smithi050 (mon.0) 229 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 69/357 objects degraded (19.328%), 19 pgs degraded" in cluster log 2024-04-09T05:52:46.364 INFO:scrape:1 jobs: ['7648047'] 2024-04-09T05:52:46.364 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-09T05:52:46.364 INFO:scrape: 2024-04-09T05:52:46.367 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/ failed: 43 dead: 0 running: 0 waiting: 0 queued: 0 passed: 40 Fail ================================================================= [7647981] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:21:29 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647981/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647981/ "2024-04-09T04:38:26.261753+0000 mon.smithi027 (mon.0) 766 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7647982] orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:16:27 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647982/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647982/ "2024-04-09T04:31:06.252687+0000 mon.a (mon.0) 386 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7647985] orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:35:25 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647985/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647985/ "2024-04-09T04:32:27.300568+0000 mon.a (mon.0) 613 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7647986] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:13:30 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647986/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647986/ "2024-04-09T04:31:37.800775+0000 mon.a (mon.0) 537 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log [7647988] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} ----------------------------------------------------------------- time: 00:27:17 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647988/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647988/ "2024-04-09T04:28:56.992890+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log [7647990] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:16:54 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647990/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647990/ "2024-04-09T04:33:11.165807+0000 mon.smithi155 (mon.0) 871 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7647991] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:11:39 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647991/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647991/ "2024-04-09T04:29:07.278176+0000 mon.smithi123 (mon.0) 239 : cluster [WRN] Health check failed: 1 host (1 osds) down (OSD_HOST_DOWN)" in cluster log [7647993] orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:48:22 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647993/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647993/ "2024-04-09T04:44:45.252512+0000 mon.a (mon.0) 936 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7647997] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} ----------------------------------------------------------------- time: 00:17:04 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647997/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647997/ "2024-04-09T04:35:06.834998+0000 mon.smithi136 (mon.0) 773 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7647998] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:18:42 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647998/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647998/ "2024-04-09T04:41:22.898598+0000 osd.1 (osd.1) 3 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running" in cluster log [7647999] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:25:34 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647999/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647999/ "2024-04-09T04:37:17.274715+0000 mon.a (mon.0) 460 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7648000] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:25:15 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648000/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648000/ "2024-04-09T04:40:53.341919+0000 mon.a (mon.0) 439 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7648001] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:14:13 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648001/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648001/ "2024-04-09T04:40:57.955341+0000 mon.smithi016 (mon.0) 788 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7648003] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:13:56 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648003/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648003/ "2024-04-09T04:45:40.238532+0000 mon.smithi005 (mon.0) 791 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648007] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:37 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648007/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648007/ "2024-04-09T04:52:17.808565+0000 mon.smithi138 (mon.0) 822 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7648008] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:19:54 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648008/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648008/ "2024-04-09T04:54:55.789675+0000 mon.smithi006 (mon.0) 553 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648012] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:42:05 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648012/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648012/ "2024-04-09T04:56:33.556166+0000 mon.a (mon.0) 432 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7648013] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:17:04 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648013/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648013/ "2024-04-09T04:56:02.536737+0000 mon.a (mon.0) 377 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7648017] orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:15:58 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648017/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648017/ "2024-04-09T04:56:35.108954+0000 mon.a (mon.0) 652 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7648021] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:53:21 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648021/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648021/ reached maximum tries (51) after waiting for 300 seconds [7648024] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} ----------------------------------------------------------------- time: 00:14:26 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648024/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648024/ "2024-04-09T04:57:16.346370+0000 mon.smithi012 (mon.0) 571 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648025] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:21:39 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648025/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648025/ "2024-04-09T05:04:53.349327+0000 mon.smithi081 (mon.0) 803 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7648026] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:20:03 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648026/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648026/ "2024-04-09T05:05:58.370400+0000 osd.1 (osd.1) 3 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running" in cluster log [7648030] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:14:34 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648030/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648030/ "2024-04-09T05:01:58.732036+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log [7648034] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:19:53 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648034/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648034/ "2024-04-09T05:08:29.684906+0000 mon.smithi018 (mon.0) 484 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648035] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:11:56 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648035/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648035/ "2024-04-09T05:03:14.283291+0000 mon.a (mon.0) 473 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7648036] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:15:14 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648036/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648036/ "2024-04-09T05:06:16.221710+0000 mon.smithi045 (mon.0) 784 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648038] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:14:18 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648038/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648038/ "2024-04-09T05:04:12.491347+0000 mon.a (mon.0) 349 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7648039] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:22:03 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648039/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648039/ "2024-04-09T05:10:33.258156+0000 mon.smithi062 (mon.0) 844 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648040] orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:27 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648040/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648040/ "2024-04-09T05:04:56.460115+0000 osd.1 (osd.1) 3 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running" in cluster log [7648041] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:15:41 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648041/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648041/ "2024-04-09T05:04:15.477415+0000 mon.smithi123 (mon.0) 807 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648045] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} ----------------------------------------------------------------- time: 00:16:54 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648045/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648045/ "2024-04-09T05:07:17.399425+0000 mon.smithi080 (mon.0) 778 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648047] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:56 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648047/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648047/ "2024-04-09T05:20:00.000126+0000 mon.smithi050 (mon.0) 229 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 69/357 objects degraded (19.328%), 19 pgs degraded" in cluster log [7648048] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:14:52 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648048/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648048/ "2024-04-09T05:11:57.790831+0000 mon.smithi149 (mon.0) 716 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log [7648050] orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:25:34 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648050/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648050/ "2024-04-09T05:14:48.028214+0000 mon.a (mon.0) 408 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7648051] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} ----------------------------------------------------------------- time: 00:47:12 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648051/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648051/ "2024-04-09T05:31:24.328647+0000 mon.a (mon.0) 1450 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7648054] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} ----------------------------------------------------------------- time: 00:15:42 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648054/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648054/ "2024-04-09T05:20:31.573392+0000 mon.smithi043 (mon.0) 794 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log [7648056] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} ----------------------------------------------------------------- time: 00:19:52 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648056/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648056/ "2024-04-09T05:26:08.728344+0000 mon.smithi099 (mon.0) 754 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648057] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:19:58 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648057/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648057/ "2024-04-09T05:24:10.936403+0000 mon.a (mon.0) 455 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7648058] orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:20:50 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648058/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648058/ "2024-04-09T05:26:02.825299+0000 mon.smithi105 (mon.0) 796 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log [7648059] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:39:08 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648059/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648059/ reached maximum tries (51) after waiting for 300 seconds [7648060] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} ----------------------------------------------------------------- time: 00:14:18 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648060/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648060/ "2024-04-09T05:22:21.033328+0000 mon.smithi005 (mon.0) 786 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7648061] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} ----------------------------------------------------------------- time: 00:18:25 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648061/ log: http://qa-proxy.ceph.com/teuthology/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648061/ "2024-04-09T05:21:09.400888+0000 mon.smithi037 (mon.0) 707 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log Pass ================================================================= [7647983] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:02 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647983/ [7647984] orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:34:59 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647984/ [7647987] orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} time: 00:14:31 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647987/ [7647989] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} time: 00:13:35 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647989/ [7647992] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:11:39 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647992/ [7647994] orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} time: 00:39:20 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647994/ [7647995] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:23:28 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647995/ [7647996] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:14:34 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7647996/ [7648002] orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} time: 00:24:45 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648002/ [7648004] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} time: 00:49:22 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648004/ [7648005] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:38:49 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648005/ [7648006] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} time: 00:13:47 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648006/ [7648009] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:24 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648009/ [7648010] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:18:06 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648010/ [7648011] orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:12:26 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648011/ [7648014] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} time: 00:12:56 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648014/ [7648015] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} time: 00:13:52 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648015/ [7648016] orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} time: 00:19:36 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648016/ [7648018] orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:18:52 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648018/ [7648019] orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 00:52:07 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648019/ [7648020] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:45:31 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648020/ [7648022] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:13:56 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648022/ [7648023] orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} time: 00:19:33 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648023/ [7648027] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:22:59 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648027/ [7648028] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} time: 00:06:35 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648028/ [7648029] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:31 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648029/ [7648031] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} time: 00:12:33 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648031/ [7648032] orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} time: 00:11:08 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648032/ [7648033] orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:14:38 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648033/ [7648037] orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:43 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648037/ [7648042] orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:51:04 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648042/ [7648043] orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys} time: 00:25:42 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648043/ [7648044] orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} time: 00:14:16 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648044/ [7648046] orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:13:20 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648046/ [7648049] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:39:00 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648049/ [7648052] orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:44:16 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648052/ [7648053] orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} time: 00:20:38 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648053/ [7648055] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} time: 00:37:42 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648055/ [7648062] orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:23:07 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648062/ [7648063] orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:16:07 info: https://pulpito.ceph.com/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648063/