Found 131 jobs
Missing teuthology log /home/teuthworker/archive/yuriw-2022-11-23_17:41:56-orch-main-distro-default-smithi/7089391/teuthology.log
Found 10 distinct failure reasons
Failure: Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
1 jobs: ['7089361']
suites: ['agent/on', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_orch_cli}']

Dead: None
16 jobs: ['7089391', '7089386', '7089405', '7089346', '7089406', '7089353', '7089327', '7089363', '7089301', '7089328', '7089307', '7089325', '7089317', '7089362', '7089312', '7089300']

Failure: SELinux denials found on ubuntu@smithi081.front.sepia.ceph.com: ['type=AVC msg=audit(1669605573.282:19305): avc:  denied  { ioctl } for  pid=113130 comm="iptables" path="/var/lib/containers/storage/overlay/61fccfedb97779a2cc7295e8cab380961e6b100953a18bec200649ef4fa325c4/merged" dev="overlay" ino=3540444 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
1 jobs: ['7089381']
suites: ['agent/off', 'mon_election/classic', 'orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_cephadm}']

Failure: Command failed on smithi155 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.155:/foouser /mnt/foo'"
11 jobs: ['7089333', '7089378', '7089306', '7089339', '7089382', '7089303', '7089394', '7089299', '7089302', '7089334', '7089357']
suites intersection: []
suites union: ['0-nvme-loop', '1-ranks/2', '1-start', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-services/nfs-ingress', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '2-services/nfs-ingress2', '2-services/rgw', '2-thrash', '3-final}', '3-inline/yes', '3-tasks/rados_api_tests', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'agent/off', 'agent/on', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fixed-2', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async-v2only', 'orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools', 'orch/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/smoke-roleless/{0-distro/ubuntu_20.04', 'orch/cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/thrash/{0-distro/ubuntu_20.04', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'root}', 'start}', 'syntax}', 'tasks/{0-from/pacific']

Failure: SELinux denials found on ubuntu@smithi112.front.sepia.ceph.com: ['type=AVC msg=audit(1669602212.771:19378): avc:  denied  { ioctl } for  pid=120324 comm="iptables" path="/var/lib/containers/storage/overlay/3949c31c8b9b98917048da392236cba26f7ffae5207ba7b5844cd476e7e5b3f2/merged" dev="overlay" ino=3412539 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1', 'type=AVC msg=audit(1669602212.812:19379): avc:  denied  { ioctl } for  pid=120328 comm="iptables" path="/var/lib/containers/storage/overlay/3949c31c8b9b98917048da392236cba26f7ffae5207ba7b5844cd476e7e5b3f2/merged" dev="overlay" ino=3412539 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
1 jobs: ['7089329']
suites: ['agent/on', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'task/test_cephadm}']

Failure: 'check osd count' reached maximum tries (90) after waiting for 900 seconds
2 jobs: ['7089413', '7089347']
suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}']
suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/flannel', 'net/host', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}']

Failure: Command failed on smithi094 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f0134ae-6ec9-11ed-843b-001a4aab830c -e sha1=697b0d99d9a52df92b20c8e053dcd46c75f364a9 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 2\'"\'"\'\''
3 jobs: ['7089377', '7089337', '7089313']
suites intersection: ['3-upgrade/staggered', '4-wait', '5-upgrade-ls']
suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade/staggered', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04']

Failure: {'smithi093.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['rpm', '-e', 'ceph-debuginfo'], 'delta': '0:00:00.012412', 'end': '2022-11-28 01:48:31.799017', 'invocation': {'module_args': {'_raw_params': 'rpm -e ceph-debuginfo', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-11-28 01:48:31.786605', 'stderr': 'error: Failed dependencies:\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-radosgw-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) libcephfs2-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) rbd-mirror-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-base-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-osd-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) python3-rbd-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-common-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-mgr-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-fuse-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) librbd1-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) python3-rgw-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-immutable-object-cache-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-mds-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) librados-devel-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) python3-cephfs-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) rbd-fuse-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-mon-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) python3-rados-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-test-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) librados2-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64\n\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) rbd-nbd-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', 'stderr_lines': ['error: Failed dependencies:', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-radosgw-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) libcephfs2-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) rbd-mirror-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-base-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-osd-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) python3-rbd-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-common-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-mgr-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-fuse-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) librbd1-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) python3-rgw-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-immutable-object-cache-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-mds-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) librados-devel-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) python3-cephfs-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) rbd-fuse-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-mon-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) python3-rados-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) ceph-test-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) librados2-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64', '\tceph-debuginfo(x86-64) = 2:18.0.0-594.g697b0d99.el8 is needed by (installed) rbd-nbd-debuginfo-2:18.0.0-594.g697b0d99.el8.x86_64'], 'stdout': '', 'stdout_lines': []}}
1 jobs: ['7089298']
suites: ['1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools']

Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)
2 jobs: ['7089351', '7089295']
suites intersection: ['agent/off', 'task/test_nfs}']
suites union: ['agent/off', 'mon_election/classic', 'mon_election/connectivity', 'orch/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'orch/cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_nfs}']

Failure: Command failed on smithi146 with status 1: 'kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml'
2 jobs: ['7089314', '7089380']
suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}']
suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'orch/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}']