Found 305 jobs
Found 14 distinct failure reasons
Dead: b'2023-03-19T20:40:38.191 INFO:teuthology.orchestra.run.smithi029.stdout:            ],'
1 jobs: ['7212241']
suites: ['debug/mgr', 'mgr_ttl_cache/disable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-hybrid}', 'supported-random-distro$/{centos_8}', 'tasks/insights}']

Dead: b'2023-03-19T23:18:38.739 INFO:journalctl@ceph.osd.9.smithi159.stdout:Mar 19 16:36:56 smithi159 ceph-ad128aca-c649-11ed-9afc-001a4aab830c-osd-9[119116]: 2023-03-19T16:36:56.097+0000 7f85b1117700 -1 osd.9 78 heartbeat_check: no reply from 172.21.15.44:6807 osd.1 ever on either front or back, first ping sent 2023-03-19T16:27:46.488034+0000 (oldest deadline 2023-03-19T16:28:06.488034+0000)'
3 jobs: ['7212386', '7212146', '7212401']
suites intersection: ['mon_election/classic']
suites union: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/nautilus-v1only', 'agent/off', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'rados', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'task/test_orch_cli_mon}', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/test_rbd_api}']

Dead: b'2023-03-19T21:12:31.376 DEBUG:teuthology.exit:Finished running handlers'
9 jobs: ['7212256', '7212329', '7212277', '7212181', '7212209', '7212275', '7212174', '7212342', '7212113']
suites intersection: []
suites union: ['0-size-min-size-overrides/3-size-2-min-size', '0-start', '1-install/octopus', '1-install/quincy', '1-tasks', 'agent/on', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'rados/dashboard/{0-single-container-host', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools}', 'rados_api', 'rados_loadgenbig', 'random-objectstore$/{bluestore-bitmap}', 'random-objectstore$/{bluestore-comp-lz4}', 'random-objectstore$/{bluestore-comp-snappy}', 'rbd_import_export', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'task/test_orch_cli_mon}', 'tasks/dashboard}', 'tasks/failover}', 'tasks/module_selftest}', 'tasks/prometheus}', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/default', 'thrashers/morepggrow', 'thrashosds-health', 'three-plus-one}', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/radosbench}', 'workloads/snaps-few-objects}']

Failure: SELinux denials found on ubuntu@smithi100.front.sepia.ceph.com: ['type=AVC msg=audit(1679198508.680:19826): avc:  denied  { ioctl } for  pid=109514 comm="iptables" path="/var/lib/containers/storage/overlay/39edf9328336b455ef9006c03b64ed46d41c81c9ee2181709eb11f5158b4a32f/merged" dev="overlay" ino=3278938 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
1 jobs: ['7212184']
suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_cephadm}']

Failure: Command failed on smithi139 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull'
5 jobs: ['7212178', '7212408', '7212358', '7212330', '7212254']
suites intersection: []
suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'agent/on', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'mon_election/connectivity', 'net/calico', 'net/flannel', 'net/host', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}', 'task/test_iscsi_pids_limit/{centos_8.stream_container_tools', 'test_iscsi_pids_limit}}']

Failure: No module named 'tasks.ceph'
2 jobs: ['7212402', '7212407']
suites intersection: []
suites union: ['debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/singleton/{all/deduptool', 'random-objectstore$/{bluestore-comp-snappy}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/crash}']

Failure: Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi037 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6dd66830c12498d02f3894e352dd8b49a7bab4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh'
2 jobs: ['7212384', '7212336']
suites intersection: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'rados', 'test_envlibrados_for_rocksdb}']
suites union: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb/{supported/centos_latest', 'rados/singleton/{all/test_envlibrados_for_rocksdb/{supported/rhel_latest', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'test_envlibrados_for_rocksdb}']

Failure: failed to complete snap trimming before timeout
1 jobs: ['7212332']
suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']

Failure: Command failed on smithi039 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-2743-ga6dd6683-1focal ceph-mds=18.0.0-2743-ga6dd6683-1focal ceph-common=18.0.0-2743-ga6dd6683-1focal ceph-fuse=18.0.0-2743-ga6dd6683-1focal ceph-test=18.0.0-2743-ga6dd6683-1focal radosgw=18.0.0-2743-ga6dd6683-1focal python-ceph=18.0.0-2743-ga6dd6683-1focal libcephfs1=18.0.0-2743-ga6dd6683-1focal libcephfs-java=18.0.0-2743-ga6dd6683-1focal libcephfs-jni=18.0.0-2743-ga6dd6683-1focal librados2=18.0.0-2743-ga6dd6683-1focal librbd1=18.0.0-2743-ga6dd6683-1focal rbd-fuse=18.0.0-2743-ga6dd6683-1focal'
1 jobs: ['7212405']
suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-small-objects}']

Dead: b'2023-03-19T17:08:30.532 INFO:journalctl@ceph.osd.7.smithi155.stdout:Mar 19 08:39:25 smithi155 ceph-5bc441b0-c615-11ed-9afc-001a4aab830c-osd-7[128487]: 2023-03-19T08:39:25.901+0000 7fbc0ab73700 -1 osd.7 1841 heartbeat_check: no reply from 172.21.15.100:6814 osd.1 ever on either front or back, first ping sent 2023-03-19T08:30:43.465270+0000 (oldest deadline 2023-03-19T08:31:03.465270+0000)'
1 jobs: ['7212213']
suites: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/nautilus', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashers/careful', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}']

Failure: Timeout opening channel.
2 jobs: ['7212125', '7212154']
suites intersection: ['rados']
suites union: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/nautilus-v2only', 'backoff/normal', 'ceph', 'clusters/9-mons', 'clusters/{openstack', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/monthrash/{ceph', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/many', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/rados_mon_workunits}', 'workloads/snaps-few-objects}']

Timeout 3h running clone.client.0/qa/workunits/mon/test_mon_osdmap_prune.sh
2 jobs: ['7212121', '7212324']
suites intersection: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'workloads/rados_mon_osdmap_prune}']
suites union: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'thrashers/force-sync-many', 'thrashers/many', 'workloads/rados_mon_osdmap_prune}']

Failure: Command failed (workunit test mon/caps.sh) on smithi179 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6dd66830c12498d02f3894e352dd8b49a7bab4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh'
1 jobs: ['7212359']
suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/one', 'workloads/rados_mon_workunits}']

Failure: Command failed (workunit test mon/mon-last-epoch-clean.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6dd66830c12498d02f3894e352dd8b49a7bab4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-last-epoch-clean.sh'
1 jobs: ['7212349']
suites: ['rados/standalone/{supported-random-distro$/{centos_8}', 'workloads/mon}']