Found 94 jobs Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Found 12 distinct failure reasons Failure: "/var/log/ceph/3bdec0c2-661b-11ee-8db5-212e2dc638e7/ceph-mon.a.log:2023-10-08T20:45:51.285+0000 7f6f58471700 0 log_channel(cluster) log [WRN] : mon.d (rank 3) addr [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] is down (out of quorum)" in cluster log 37 jobs suites intersection: [] suites union: ['0-nvme-loop', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-ops/rm-zap-flag', '2-ops/rmdir-reactivate', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-services/client-keyring', '2-services/nfs', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-user', '2-services/nfs-keepalive-only', '2-thrash', '3-final', '3-inline/no', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'base/install', 'centos_8.stream_container_tools', 'cluster/{fixed-3', 'conf/{client', 'conf/{disable-pool-app}', 'fail_fs/no', 'fail_fs/yes', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'openstack}', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/nfs/{cluster/{1-node}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke/{0-distro/ubuntu_20.04', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'osd}', 'overrides/ignorelist', 'overrides/ignorelist_health', 'overrides/ignorelist}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'root}', 'start', 'start}', 'supported-random-distros$/{ubuntu_20.04}', 'syntax}', 'task/test_ca_signed_key}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'tasks/nfs}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'tasks/{0-from/quincy', 'workloads/cephadm_iscsi}'] Failure: "/var/log/ceph/a6cb7930-661f-11ee-8db5-212e2dc638e7/ceph-mon.a.log:2023-10-08T21:25:30.360+0000 7f7e25846700 0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 69/630 objects degraded (10.952%), 23 pgs degraded (PG_DEGRADED)" in cluster log 5 jobs: ['7416053', '7415995', '7416010', '7416037', '7416040'] suites intersection: [] suites union: ['0-nvme-loop', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-repo_digest/repo_digest', '2-services/nfs-haproxy-proto', '2-thrash', '3-final', '3-inline/yes', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade/simple', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic', 'msgr/async', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'osd}', 'overrides/ignorelist', 'overrides/ignorelist}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'root}', 'start}', 'syntax}', 'tasks/{0-from/quincy'] Failure: reached maximum tries (301) after waiting for 300 seconds 1 jobs: ['7416031'] suites: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'overrides/ignorelist}'] Failure: "/var/log/ceph/7d51a076-661e-11ee-8db5-212e2dc638e7/ceph-mon.smithi173.log:2023-10-08T21:18:55.740+0000 7f1a7d52f700 0 log_channel(cluster) log [WRN] : Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log 5 jobs: ['7416047', '7415997', '7416042', '7416002', '7416054'] suites intersection: [] suites union: ['0-nvme-loop', '1-ranks/1', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-services/nfs-ingress-rgw-user', '2-services/nfs-keepalive-only', '3-final', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/root', 'mon', 'mon_election/connectivity', 'msgr/async', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'osd}', 'overrides/ignorelist', 'overrides/ignorelist}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'start', 'syntax}', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'tasks/rados_api_tests}', 'tasks/{0-from/quincy', 'test_iscsi_container}}'] Failure: "/var/log/ceph/d7fa8358-661e-11ee-8db5-212e2dc638e7/ceph-mon.c.log:2023-10-08T21:20:00.750+0000 7f99ec7f3700 7 mon.c@2(peon).log v488 update_from_paxos applying incremental log 488 2023-10-08T21:20:00.000125+0000 mon.a (mon.0) 1117 : cluster [WRN] application not enabled on pool 'rbd'" in cluster log 4 jobs: ['7416048', '7416020', '7416030', '7415986'] suites intersection: ['fixed-2', 'overrides/ignorelist'] suites union: ['1-start', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'overrides/ignorelist', 'root}', 'start', 'tasks/rados_python}'] Failure: "/var/log/ceph/1c000438-661a-11ee-8db5-212e2dc638e7/ceph-mon.c.log:2023-10-08T20:49:07.256+0000 7fc56fa20700 7 mon.c@2(peon).log v584 update_from_paxos applying incremental log 584 2023-10-08T20:49:06.379940+0000 mgr.x (mgr.24487) 29 : cephadm [ERR] Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-x/keyring', '-n', 'mgr.x', 'tell', 'mgr.x', 'rotate-key', '-i', '-']: 2023-10-08T20:49:06.373+0000 7f1cfd553700 1 -- 172.21.15.79:0/3424155995 <== mon.1 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7f1ce4003770 con 0x7f1cf8109960" in cluster log 2 jobs: ['7416013', '7416036'] suites intersection: ['fixed-2', 'msgr/async-v2only', 'overrides/ignorelist', 'start', 'tasks/rotate-keys}'] suites union: ['fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async-v2only', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'overrides/ignorelist', 'start', 'tasks/rotate-keys}'] Failure: "/var/log/ceph/ad496f74-661a-11ee-8db5-212e2dc638e7/ceph-mon.a.log:2023-10-08T20:49:59.999+0000 7f25968dd700 0 log_channel(cluster) log [WRN] : application not enabled on pool 'ceph_test_rados_api_asio'" in cluster log 2 jobs: ['7415999', '7416065'] suites intersection: [] suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/root', 'mon', 'mon_election/connectivity', 'msgr/async', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'osd}', 'overrides/ignorelist', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'start', 'syntax}', 'tasks/rados_api_tests}', 'tasks/{0-from/quincy'] Failure: "/var/log/ceph/7e64fed8-661c-11ee-8db5-212e2dc638e7/ceph-mon.c.log:2023-10-08T21:00:00.452+0000 7f3bc6a83700 7 mon.c@2(peon).log v327 update_from_paxos applying incremental log 327 2023-10-08T21:00:00.003048+0000 mon.a (mon.0) 1166 : cluster [WRN] [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log 1 jobs: ['7416021'] suites: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'overrides/ignorelist', 'start', 'tasks/rados_api_tests}'] Failure: "/var/log/ceph/42336e2a-661e-11ee-8db5-212e2dc638e7/ceph-mon.c.log:2023-10-08T21:13:02.669+0000 7f3377f33700 7 mon.c@2(peon).log v306 update_from_paxos applying incremental log 306 2023-10-08T21:13:01.986623+0000 mon.a (mon.0) 1014 : cluster [WRN] Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN)" in cluster log 1 jobs: ['7416045'] suites: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/classic', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'overrides/ignorelist', 'start}'] Failure: Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 718f9526-661f-11ee-8db5-212e2dc638e7 -- ceph mon dump -f json' 1 jobs: ['7416049'] suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] Failure: "/var/log/ceph/fc8bed62-661c-11ee-8db5-212e2dc638e7/ceph-mon.smithi092.log:2023-10-08T21:05:45.398+0000 7fcc35a3d700 10 mon.smithi092@0(leader).log v338 logging 2023-10-08T21:05:44.609614+0000 osd.1 (osd.1) 3 : cluster [WRN] Monitor daemon marked osd.1 down, but it is still running" in cluster log 1 jobs: ['7416033'] suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add', 'orch:cephadm/osds/{0-distro/ubuntu_20.04', 'overrides/ignorelist}'] Failure: Command failed on smithi064 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 96bdfb64-6618-11ee-8db5-212e2dc638e7 -e sha1=54d669b34dfdbcc2574331a0352cd10651c426ed -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 1 jobs: ['7415979'] suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'overrides/ignorelist}']