Found 98 jobs Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Found 16 distinct failure reasons Failure: "/var/log/ceph/04c46944-c123-11ee-95b6-87774f69a715/ceph-mon.a.log:2024-02-01T17:03:28.092+0000 7fed000d4700 0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 30 jobs suites intersection: [] suites union: ['0-nvme-loop', '1-start', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/snaps-few-objects', '3-upgrade/simple', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke/{0-distro/ubuntu_20.04', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'root}', 'start', 'start}', 'task/test_ca_signed_key}', 'task/test_extra_daemon_features}', 'task/test_host_drain}', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/rotate-keys}', 'test_iscsi_container}}'] Failure: Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI) 1 jobs: ['7541702'] suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_orch_cli}'] Failure: "/var/log/ceph/1f62c544-c127-11ee-95b6-87774f69a715/ceph-mon.smithi052.log:2024-02-01T17:30:45.441+0000 7f6fff28e700 10 mon.smithi052@0(leader).log v100 logging 2024-02-01T17:30:45.148062+0000 mgr.smithi052.yesuit (mgr.14217) 15 : cephadm [ERR] Failed while placing ceph-exporter.smithi067 on smithi067: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-1f62c544-c127-11ee-95b6-87774f69a715-ceph-exporter-smithi067" in cluster log 28 jobs: ['7541677', '7541705', '7541567', '7541645', '7541582', '7541553', '7541696', '7541611', '7541576', '7541535', '7541641', '7541657', '7541630', '7541690', '7541603', '7541571', '7541614', '7541688', '7541594', '7541663', '7541654', '7541618', '7541700', '7541538', '7541666', '7541598', '7541587', '7541625'] suites intersection: ['0-nvme-loop', '1-start'] suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/mirror', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '3-final}', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/osds/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] Failure: "/var/log/ceph/74632256-c126-11ee-95b6-87774f69a715/ceph-mon.smithi106.log:2024-02-01T17:26:44.261+0000 7f8b515e6700 10 mon.smithi106@0(leader).log v109 logging 2024-02-01T17:26:43.717624+0000 mgr.smithi106.rljkgl (mgr.14195) 17 : cephadm [ERR] Failed while placing ceph-exporter.smithi118 on smithi118: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-74632256-c126-11ee-95b6-87774f69a715-ceph-exporter-smithi118" in cluster log 2 jobs: ['7541670', '7541560'] suites intersection: ['0-nvme-loop', '1-start', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-user', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] Failure: "/var/log/ceph/2d2e8902-c121-11ee-95b6-87774f69a715/ceph-mon.a.log:2024-02-01T16:53:38.166+0000 7fc549b40700 0 log_channel(cluster) log [WRN] : Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log 8 jobs: ['7541578', '7541621', '7541703', '7541609', '7541616', '7541636', '7541674', '7541620'] suites intersection: [] suites union: ['0-nvme-loop', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-thrash', '3-inline/yes', '3-tasks/rados_api_tests', '3-tasks/small-objects', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/packaged', 'mon', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'root}', 'start', 'start}', 'syntax}', 'task/test_host_drain}', 'task/test_set_mon_crush_locations}', 'tasks/rados_api_tests}', 'tasks/{0-from/quincy'] Failure: "/var/log/ceph/2f5865de-c128-11ee-95b6-87774f69a715/ceph-mon.a.log:2024-02-01T17:39:03.386+0000 7f13e4f0c700 0 log_channel(cluster) log [WRN] : Health check failed: 1/5 mons down, quorum a,e,c,d (MON_DOWN)" in cluster log 5 jobs: ['7541707', '7541629', '7541659', '7541548', '7541569'] suites intersection: [] suites union: ['1-start', '2-repo_digest/repo_digest', '2-thrash', '3-tasks/small-objects', '3-upgrade/simple', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'base/install', 'cluster/{fixed-3', 'conf/{disable-pool-app}', 'fixed-2', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr/async', 'msgr/async-v1only', 'openstack}', 'orch:cephadm/rbd_iscsi/{0-single-container-host', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'root}', 'start', 'task/test_orch_cli_mon}', 'tasks/rados_python}', 'workloads/cephadm_iscsi}'] Failure: "/var/log/ceph/bf4f5ff6-c125-11ee-95b6-87774f69a715/ceph-mon.a.log:2024-02-01T17:21:00.956+0000 7fe4135c1700 0 log_channel(cluster) log [WRN] : Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log 2 jobs: ['7541668', '7541652'] suites intersection: [] suites union: ['0-nvme-loop', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'start}', 'syntax}', 'tasks/{0-from/quincy'] Failure: "/var/log/ceph/4ff59fca-c121-11ee-95b6-87774f69a715/ceph-mon.smithi001.log:2024-02-01T16:54:11.284+0000 7f6c0934d700 0 log_channel(cluster) log [WRN] : Replacing daemon mds.cephfs.smithi001.tdgzwo as rank 0 with standby daemon mds.cephfs.smithi001.gjemon" in cluster log 2 jobs: ['7541612', '7541585'] suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] Failure: "/var/log/ceph/63f8e936-c127-11ee-95b6-87774f69a715/ceph-mon.a.log:2024-02-01T17:42:59.907+0000 7fd82a3b0700 10 mon.a@0(leader).log v528 logging 2024-02-01T17:42:59.400762+0000 osd.2 (osd.2) 3 : cluster [WRN] failed to encode map e105 with expected crc" in cluster log 2 jobs: ['7541683', '7541639'] suites intersection: [] suites union: ['1-start', '2-repo_digest/repo_digest', '2-thrash', '3-tasks/radosbench', '3-upgrade/simple', '4-wait', '5-upgrade-ls', 'agent/off', 'fixed-2', 'mon_election/classic}', 'msgr/async-v2only', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'root}'] Failure: "/var/log/ceph/59ed585a-c127-11ee-95b6-87774f69a715/ceph-mon.smithi096.log:2024-02-01T17:44:23.600+0000 7f08ca7b1700 10 mon.smithi096@0(leader).log v589 logging 2024-02-01T17:44:23.330526+0000 osd.3 (osd.3) 3 : cluster [WRN] failed to encode map e48 with expected crc" in cluster log 1 jobs: ['7541676'] suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] Failure: Command failed on smithi158 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5'" 1 jobs: ['7541542'] suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] Failure: "/var/log/ceph/9cc7adb2-c125-11ee-95b6-87774f69a715/ceph-mon.a.log:2024-02-01T17:23:08.489+0000 7f9648068700 10 mon.a@0(leader).log v198 logging 2024-02-01T17:23:07.981950+0000 mgr.a (mgr.14150) 169 : cephadm [ERR] Task exception was never retrieved" in cluster log 1 jobs: ['7541656'] suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm_timeout}'] Failure: ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6252 (apt-get)'] 1 jobs: ['7541694'] suites: ['agent/off', 'mon_election/classic', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_monitoring_stack_basic}'] Failure: "/var/log/ceph/40db7518-c11d-11ee-95b6-87774f69a715/ceph-mon.a.log:2024-02-01T16:21:27.036+0000 7fe72ea1a700 10 mon.a@0(leader).log v172 logging 2024-02-01T16:21:26.266841+0000 mgr.x (mgr.14310) 20 : cephadm [WRN] Bind address in nfs.test.0.0.smithi175.ifiomy's ganesha conf is defaulting to empty" in cluster log 1 jobs: ['7541544'] suites: ['conf/{client', 'mds', 'mon', 'orch:cephadm/nfs/{cluster/{1-node}', 'osd}', 'overrides/ignorelist_health', 'supported-random-distros$/{centos_latest}', 'tasks/nfs}'] Failure: "/var/log/ceph/b9a36856-c128-11ee-95b6-87774f69a715/ceph-mon.a.log:2024-02-01T17:51:56.293+0000 7f71986b9700 10 mon.a@0(leader).log v550 logging 2024-02-01T17:51:55.510880+0000 mgr.x (mgr.24499) 28 : cephadm [ERR] Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-x/keyring', '-n', 'mgr.x', 'tell', 'mgr.x', 'rotate-key', '-i', '-']: 2024-02-01T17:51:55.504+0000 7f8935085700 1 -- 172.21.15.70:0/3067764102 <== mon.1 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7f891c003770 con 0x7f893010d6b0" in cluster log 1 jobs: ['7541686'] suites: ['fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr/async-v2only', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'start', 'tasks/rotate-keys}'] Failure: "/var/log/ceph/fc595f1a-c11f-11ee-95b6-87774f69a715/ceph-mon.smithi031.log:2024-02-01T16:44:09.698+0000 7f5d710e9700 0 log_channel(cluster) log [WRN] : Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log 1 jobs: ['7541558'] suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']