Found 92 jobs Missing teuthology log /home/teuthworker/archive/adking-2023-06-28_15:59:19-orch:cephadm-wip-adk3-testing-2023-06-27-0854-distro-default-smithi/7320806/teuthology.log Found 11 distinct failure reasons Dead: None 53 jobs Failure: "/var/log/ceph/d23daaf4-15f0-11ee-9b2e-001a4aab830c/ceph-mon.a.log:2023-06-28T20:19:58.203+0000 7f404b39a700 0 log_channel(cluster) log [WRN] : Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log 18 jobs: ['7320770', '7320831', '7320842', '7320848', '7320826', '7320830', '7320777', '7320776', '7320810', '7320814', '7320763', '7320786', '7320827', '7320845', '7320791', '7320772', '7320768', '7320852'] suites intersection: [] suites union: ['0-nvme-loop', '1-start', '2-ops/rm-zap-flag}', '2-ops/rmdir-reactivate}', '2-services/nfs-ingress2', '2-services/rgw', '2-thrash', '3-final}', '3-tasks/small-objects', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'openstack}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'root}', 'start', 'start}', 'task/test_nfs}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'workloads/cephadm_iscsi}'] Failure: "/var/log/ceph/ac9b5c5e-15f2-11ee-9b2e-001a4aab830c/ceph-mon.smithi071.log:2023-06-28T20:39:14.164+0000 7f83fd67c700 10 mon.smithi071@0(leader).log v317 logging 2023-06-28T20:39:13.191877+0000 mgr.smithi193.tkuffu (mgr.24347) 87 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin" in cluster log 2 jobs: ['7320782', '7320797'] suites intersection: [] suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-repo_digest/defaut', '3-inline/no', '3-upgrade-mgr-staggered', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] Failure: "/var/log/ceph/af541ef6-15f0-11ee-9b2e-001a4aab830c/ceph-mon.smithi084.log:2023-06-28T20:24:18.832+0000 7fb31819e700 10 mon.smithi084@0(leader).log v284 logging 2023-06-28T20:24:18.600826+0000 mgr.smithi086.exikoe (mgr.14660) 37 : cephadm [WRN] Bind address in nfs.foo.0.0.smithi084.doxfpe's ganesha conf is defaulting to empty" in cluster log 3 jobs: ['7320766', '7320787', '7320825'] suites intersection: ['1-start'] suites union: ['0-nvme-loop', '1-bootstrap/16.2.0', '1-start', '2-nfs', '2-services/nfs', '2-services/nfs-haproxy-proto', '3-final}', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8'] Failure: "/var/log/ceph/7fc13e8e-15f6-11ee-9b2e-001a4aab830c/ceph-mon.smithi089.log:2023-06-28T21:03:05.369+0000 7f5aabec8700 0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log 1 jobs: ['7320843'] suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun'] Failure: [Errno None] Unable to connect to port 22 on 172.21.15.62 1 jobs: ['7320824'] suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] Failure: "/var/log/ceph/6c3abb78-15f2-11ee-9b2e-001a4aab830c/ceph-mon.smithi090.log:2023-06-28T20:41:00.444+0000 7fd079dcc700 10 mon.smithi090@0(leader).log v410 logging 2023-06-28T20:40:59.967812+0000 mgr.smithi090.cllntc (mgr.14203) 299 : cephadm [ERR] Failed while placing nfs.foo.0.0.smithi090.wytzyu on smithi090: grace tool failed: rados_pool_create: -1" in cluster log 1 jobs: ['7320783'] suites: ['0-nvme-loop', '1-start', '2-services/nfs-keepalive-only', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] Failure: "/var/log/ceph/2b15c208-15f4-11ee-9b2e-001a4aab830c/ceph-mon.c.log:2023-06-28T20:49:30.805+0000 7f416a03f700 7 mon.c@2(peon).log v311 update_from_paxos applying incremental log 311 2023-06-28T20:49:29.799423+0000 mon.a (mon.0) 829 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log 3 jobs: ['7320805', '7320764', '7320774'] suites intersection: ['fixed-2'] suites union: ['0-nvme-loop', '1-start', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', 'agent/on', 'fixed-2', 'mon_election/connectivity', 'msgr/async-v1only', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'root}', 'start}'] Failure: "/var/log/ceph/5fb6d5a6-15f4-11ee-9b2e-001a4aab830c/ceph-mon.a.log:2023-06-28T20:49:59.999+0000 7f1ebb06f700 0 log_channel(cluster) log [WRN] : Health detail: HEALTH_WARN Reduced data availability: 5 pgs inactive; Degraded data redundancy: 152/594 objects degraded (25.589%), 34 pgs degraded; 1 pool(s) do not have an application enabled" in cluster log 1 jobs: ['7320788'] suites: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async-v1only', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'start', 'tasks/rados_python}'] Failure: "/var/log/ceph/13508548-15f1-11ee-9b2e-001a4aab830c/ceph-mon.smithi153.log:2023-06-28T20:25:02.972+0000 7f9417ff7700 10 mon.smithi153@0(leader).log v214 logging 2023-06-28T20:25:02.969177+0000 mgr.smithi153.hmhfwt (mgr.14227) 156 : cephadm [WRN] Bind address in nfs.foo.0.0.smithi153.hnkbgh's ganesha conf is defaulting to empty" in cluster log 1 jobs: ['7320765'] suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-bucket', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools'] Failure: "/var/log/ceph/7ca977f2-15f1-11ee-9b2e-001a4aab830c/ceph-mon.smithi007.log:2023-06-28T20:29:42.397+0000 7f05fde95700 10 mon.smithi007@0(leader).log v262 logging 2023-06-28T20:29:41.880142+0000 mgr.smithi007.cercsl (mgr.14229) 184 : cephadm [ERR] Failed while placing nfs.foo.0.0.smithi007.hqrqty on smithi007: grace tool failed: rados_pool_create: -1" in cluster log 1 jobs: ['7320775'] suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0']