Found 94 jobs
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Found 10 distinct failure reasons
Failure: "/var/log/ceph/97275b90-7b94-11ee-8db9-212e2dc638e7/ceph-mon.smithi033.log:2023-11-05T04:38:29.249+0000 7f5e0d40c700 10 mon.smithi033@0(leader).log v102  logging 2023-11-05T04:38:28.696717+0000 mgr.smithi033.minaml (mgr.14195) 17 : cephadm [ERR] Failed while placing ceph-exporter.smithi192 on smithi192: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-97275b90-7b94-11ee-8db9-212e2dc638e7-ceph-exporter-smithi192" in cluster log
29 jobs: ['7447976', '7447914', '7447932', '7447923', '7447940', '7447992', '7447972', '7447917', '7447967', '7447935', '7447986', '7447960', '7447962', '7447938', '7447995', '7447948', '7447969', '7447944', '7447929', '7447973', '7447989', '7447926', '7447985', '7447952', '7447946', '7447905', '7447955', '7447979', '7447904']
suites intersection: ['0-nvme-loop', '1-start', 'overrides/ignorelist}']
suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all', '2-ops/rm-zap-add', '2-ops/rm-zap-flag', '2-ops/rm-zap-wait', '2-ops/rmdir-reactivate', '2-services/basic', '2-services/client-keyring', '2-services/iscsi', '2-services/jaeger', '2-services/mirror', '2-services/nfs', '2-services/nfs-haproxy-proto', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/nfs-keepalive-only', '2-services/nfs2', '2-services/nvmeof', '2-services/rgw', '2-services/rgw-ingress', '3-final', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/osds/{0-distro/ubuntu_20.04', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'overrides/ignorelist}']

Failure: "/var/log/ceph/683499d0-7b92-11ee-8db9-212e2dc638e7/ceph-mon.c.log:2023-11-05T04:26:53.078+0000 7fa73b393700  7 mon.c@2(peon).log v351 update_from_paxos applying incremental log 351 2023-11-05T04:26:53.035506+0000 mon.a (mon.0) 892 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL)" in cluster log
6 jobs: ['7447959', '7447974', '7447906', '7447950', '7447949', '7447915']
suites intersection: ['fixed-2', 'overrides/ignorelist']
suites union: ['0-nvme-loop', '1-start', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/radosbench', 'agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'overrides/ignorelist', 'root}', 'start', 'start}', 'tasks/rados_api_tests}', 'tasks/rados_python}']

Failure: "/var/log/ceph/9541105c-7b92-11ee-8db9-212e2dc638e7/ceph-mon.a.log:2023-11-05T04:33:32.564+0000 7fb066d1c700 10 mon.a@0(leader).log v549  logging 2023-11-05T04:33:32.407512+0000 mgr.x (mgr.24505) 29 : cephadm [ERR] Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-x/keyring', '-n', 'mgr.x', 'tell', 'mgr.x', 'rotate-key', '-i', '-']: 2023-11-05T04:33:32.399+0000 7ff1c2ffd700  1 -- 172.21.15.46:0/2166494889 <== mon.1 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7ff1b0003770 con 0x7ff1c410cf80" in cluster log
2 jobs: ['7447965', '7447920']
suites intersection: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'overrides/ignorelist', 'start', 'tasks/rotate-keys}']
suites union: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async', 'msgr/async-v2only', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'overrides/ignorelist', 'start', 'tasks/rotate-keys}']

Failure: "/var/log/ceph/1502f21e-7b94-11ee-8db9-212e2dc638e7/ceph-mon.a.log:2023-11-05T04:41:06.957+0000 7f66b4fb7700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log
29 jobs: ['7447970', '7447980', '7447991', '7447982', '7447933', '7447941', '7447945', '7447927', '7447937', '7447934', '7447953', '7447990', '7447930', '7447981', '7447994', '7447996', '7447913', '7447936', '7447943', '7447988', '7447966', '7447918', '7447925', '7447947', '7447954', '7447922', '7447916', '7447961', '7447939']
suites intersection: []
suites union: ['0-nvme-loop', '1-ranks/1', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '2-thrash', '3-inline/no', '3-inline/yes', '3-tasks/rados_api_tests', '3-tasks/radosbench', '3-tasks/small-objects', '3-tasks/snaps-few-objects', '3-upgrade-mgr-staggered', '3-upgrade/simple', '3-upgrade/staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '4-wait', '5-upgrade-ls', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mode/packaged', 'mode/root', 'mon', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/smoke/{0-distro/ubuntu_20.04', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/thrash/{0-distro/ubuntu_20.04', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/with-work/{0-distro/ubuntu_20.04', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'osd}', 'overrides/ignorelist', 'overrides/ignorelist}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'root}', 'start', 'start}', 'syntax}', 'task/test_ca_signed_key}', 'task/test_orch_cli_mon}', 'task/test_orch_cli}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/{0-from/quincy']

Failure: "/var/log/ceph/c9a947ea-7b94-11ee-8db9-212e2dc638e7/ceph-mon.a.log:2023-11-05T04:49:59.999+0000 7f7dcbd63700  0 log_channel(cluster) log [WRN] :     application not enabled on pool 'ceph_test_rados_api_asio'" in cluster log
4 jobs: ['7447971', '7447993', '7447977', '7447928']
suites intersection: ['overrides/ignorelist']
suites union: ['agent/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v1only', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools', 'orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'overrides/ignorelist', 'start', 'task/test_orch_cli_mon}', 'tasks/rados_api_tests}', 'tasks/rados_python}']

Failure: Command failed on smithi195 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e76567e-7b95-11ee-8db9-212e2dc638e7 -- ceph mon dump -f json'
1 jobs: ['7447978']
suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']

Failure: "/var/log/ceph/ffbf8f12-7b8d-11ee-8db9-212e2dc638e7/ceph-mon.smithi003.log:2023-11-05T03:53:04.193+0000 7f6b5874d700  0 log_channel(cluster) log [WRN] : Replacing daemon mds.cephfs.smithi003.zvredt as rank 0 with standby daemon mds.cephfs.smithi153.nsuuuz" in cluster log
1 jobs: ['7447931']
suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']

Failure: "/var/log/ceph/db370f62-7b97-11ee-8db9-212e2dc638e7/ceph-mon.smithi019.log:2023-11-05T05:01:21.753+0000 7fe5ea5ba700 10 mon.smithi019@0(leader).log v101  logging 2023-11-05T05:01:20.887554+0000 mgr.smithi019.bdnvyq (mgr.14217) 15 : cephadm [ERR] Failed while placing ceph-exporter.smithi164 on smithi164: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-db370f62-7b97-11ee-8db9-212e2dc638e7-ceph-exporter-smithi164" in cluster log
2 jobs: ['7447997', '7447921']
suites intersection: ['0-nvme-loop', '1-start', 'overrides/ignorelist}']
suites union: ['0-nvme-loop', '1-start', '2-ops/rmdir-reactivate', '2-services/nfs-ingress2', '3-final', 'orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'overrides/ignorelist}']

Failure: "/var/log/ceph/4d0871de-7b8a-11ee-8db9-212e2dc638e7/ceph-mon.a.log:2023-11-05T03:24:04.338+0000 7f0907c64700 10 mon.a@0(leader).log v121  logging 2023-11-05T03:24:03.631044+0000 mgr.x (mgr.14310) 20 : cephadm [WRN] Bind address in nfs.test.0.0.smithi144.gqhrmh's ganesha conf is defaulting to empty" in cluster log
1 jobs: ['7447909']
suites: ['orch:cephadm/nfs/{cluster/{1-node}', 'overrides/ignorelist_health', 'supported-random-distros$/{centos_latest}', 'tasks/nfs}']

Failure: Command failed on smithi042 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5fb430a2-7b8a-11ee-8db9-212e2dc638e7 -e sha1=33a4e839a695206dce5ffc3db713741ee81cf420 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''
1 jobs: ['7447908']
suites: ['1-bootstrap/17.2.0', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'overrides/ignorelist}']