Found 320 jobs
Found 13 distinct failure reasons
Failure: "2023-03-18T07:10:09.288055+0000 mon.a (mon.0) 10 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log
3 jobs: ['7209418', '7209228', '7209671']
suites intersection: []
suites union: ['debug/mgr', 'mgr_ttl_cache/disable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'rados', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/singleton/{all/osd-backfill', 'rados/singleton/{all/random-eio', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'tasks/progress}']

Failure: Config file not found: "/home/teuthworker/src/github.com_ceph_ceph-c_5585a9d11893267887c9b271b6987be6325fb0bd/qa/tasks/ceph.conf.template".
1 jobs: ['7209547']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/redirect_promote_tests}']

Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi072 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5585a9d11893267887c9b271b6987be6325fb0bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'
2 jobs: ['7209625', '7209306']
suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}']
suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-bitmap}', 'random-objectstore$/{bluestore-stupid}', 'tasks/e2e}']

Dead: b'2023-03-18T05:11:54.851 DEBUG:teuthology.exit:Finished running handlers'
9 jobs: ['7209245', '7209551', '7209178', '7209208', '7209354', '7209463', '7209250', '7209358', '7209177']
suites intersection: []
suites union: ['0-nvme-loop', '0-size-min-size-overrides/2-size-2-min-size', '1-install/pacific', '1-start', '2-ops/repave-all}', 'agent/off', 'agent/on', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'rados', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8', 'rados/dashboard/{0-single-container-host', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/singleton/{all/osd-recovery', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'random-objectstore$/{bluestore-comp-snappy}', 'random-objectstore$/{bluestore-hybrid}', 'random-objectstore$/{bluestore-stupid}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'task/test_orch_cli_mon}', 'tasks/dashboard}', 'tasks/failover}', 'tasks/module_selftest}', 'thrashers/careful', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}']

Failure: Command failed on smithi171 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.5-1396-g5585a9d1-1focal ceph-mds=17.2.5-1396-g5585a9d1-1focal ceph-common=17.2.5-1396-g5585a9d1-1focal ceph-fuse=17.2.5-1396-g5585a9d1-1focal ceph-test=17.2.5-1396-g5585a9d1-1focal radosgw=17.2.5-1396-g5585a9d1-1focal python-ceph=17.2.5-1396-g5585a9d1-1focal libcephfs1=17.2.5-1396-g5585a9d1-1focal libcephfs-java=17.2.5-1396-g5585a9d1-1focal libcephfs-jni=17.2.5-1396-g5585a9d1-1focal librados2=17.2.5-1396-g5585a9d1-1focal librbd1=17.2.5-1396-g5585a9d1-1focal rbd-fuse=17.2.5-1396-g5585a9d1-1focal'
1 jobs: ['7209549']
suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}']

Dead: b'2023-03-19T05:23:16.134 INFO:journalctl@ceph.osd.6.smithi103.stdout:Mar 18 19:29:22 smithi103 ceph-53ce0764-c59a-11ed-9afc-001a4aab830c-osd-6[114784]: 2023-03-18T19:29:22.187+0000 7fc10c726700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 ever on either front or back, first ping sent 2023-03-18T19:28:38.578584+0000 (oldest deadline 2023-03-18T19:28:58.578584+0000)'
1 jobs: ['7209621']
suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_orch_cli_mon}']

Failure: failed to complete snap trimming before timeout
4 jobs: ['7209278', '7209669', '7209215', '7209469']
suites intersection: []
suites union: ['0-start', '1-tasks', 'clusters/9-mons', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'rados', 'rados/monthrash/{ceph', 'rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools}', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_rhel8}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/one', 'thrashers/sync-many', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/snaps-few-objects}']

Failure: Command failed on smithi133 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull'
4 jobs: ['7209465', '7209629', '7209212', '7209308']
suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '3-final', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04']
suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}']

Failure: Timeout opening channel.
2 jobs: ['7209198', '7209219']
suites intersection: ['mon_election/connectivity', 'rados']
suites union: ['0-size-min-size-overrides/3-size-2-min-size', '1-install/pacific', 'backoff/peering', 'ceph', 'clusters/9-mons', 'clusters/{openstack', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/monthrash/{ceph', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'supported-random-distro$/{centos_8}', 'thrashers/force-sync-many', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/rados_mon_workunits}', 'workloads/test_rbd_api}']

Dead: b'2023-03-18T05:26:39.952 INFO:teuthology.orchestra.run.smithi050.stdout:            "module_options": {'
2 jobs: ['7209282', '7209695']
suites intersection: ['debug/mgr', 'mgr_ttl_cache/disable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-zlib}', 'supported-random-distro$/{centos_8}']
suites union: ['debug/mgr', 'mgr_ttl_cache/disable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-zlib}', 'supported-random-distro$/{centos_8}', 'tasks/failover}', 'tasks/insights}']

Failure: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS)
2 jobs: ['7209274', '7209541']
suites intersection: ['task/test_nfs}']
suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8', 'task/test_nfs}']

Failure: No module named 'tasks.rados'
1 jobs: ['7209545']
suites: ['ceph', 'clusters/{fixed-2', 'fast/fast', 'mon_election/classic', 'msgr-failures/few', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-small-objects-overwrites}']

Timeout 3h running clone.client.0/qa/workunits/mon/test_mon_osdmap_prune.sh
2 jobs: ['7209539', '7209181']
suites intersection: ['rados', 'rados/monthrash/{ceph', 'workloads/rados_mon_osdmap_prune}']
suites union: ['clusters/3-mons', 'clusters/9-mons', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/many', 'thrashers/sync', 'workloads/rados_mon_osdmap_prune}']