Found 438 jobs Missing teuthology log /home/teuthworker/archive/sjust-2022-12-13_23:57:34-rados-wip-sjust-testing-2022-12-10-142339-distro-default-smithi/7115732/teuthology.log Found 11 distinct failure reasons Failure: {'smithi135.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi158.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} 5 jobs: ['7115855', '7115857', '7115867', '7115863', '7115859'] suites intersection: ['rados'] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'thrashers/careful', 'thrashers/morepggrow', 'thrashers/none', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}'] Dead: None 3 jobs: ['7115732', '7115762', '7115547'] Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 2 jobs: ['7115567', '7115746'] suites intersection: ['task/test_nfs}'] suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_nfs}'] Failure: Command failed on smithi035 with status 1: 'sudo kubeadm init --node-name smithi035 --token abcdef.enf3pxkb15v95vs2 --pod-network-cidr 10.249.16.0/21' 4 jobs: ['7115664', '7115770', '7115441', '7115551'] suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04'] suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}'] Failure: Command failed (workunit test post-file.sh) on smithi049 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8834a257b61cbfd4682b9bc3a261b65bb2468c79 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh' 2 jobs: ['7115496', '7115714'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}'] Failure: SELinux denials found on ubuntu@smithi146.front.sepia.ceph.com: ['type=AVC msg=audit(1670997271.422:19374): avc: denied { ioctl } for pid=123534 comm="iptables" path="/var/lib/containers/storage/overlay/b4ff85c0062b03281ddbbde0b9f6c85d9979944ff6185202f40dabd8d293c4d5/merged" dev="overlay" ino=3412493 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1', 'type=AVC msg=audit(1670997271.452:19375): avc: denied { ioctl } for pid=123537 comm="iptables" path="/var/lib/containers/storage/overlay/b4ff85c0062b03281ddbbde0b9f6c85d9979944ff6185202f40dabd8d293c4d5/merged" dev="overlay" ino=3412493 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] 1 jobs: ['7115856'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] Failure: SELinux denials found on ubuntu@smithi008.front.sepia.ceph.com: ['type=AVC msg=audit(1670982596.311:19380): avc: denied { ioctl } for pid=120421 comm="iptables" path="/var/lib/containers/storage/overlay/ecc87e7365009c17ccb7898646e42a07ce8df48eed013e642c77c71c6cab233b/merged" dev="overlay" ino=3412496 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] 1 jobs: ['7115480'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'task/test_cephadm}'] Failure: reached maximum tries (800) after waiting for 4800 seconds 1 jobs: ['7115571'] suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-radosbench}'] Failure: Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) 1 jobs: ['7115595'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_orch_cli}'] Crash: Command failed (workunit test rados/test_crash.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8834a257b61cbfd4682b9bc3a261b65bb2468c79 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh' ceph version 18.0.0-1386-g8834a257 (8834a257b61cbfd4682b9bc3a261b65bb2468c79) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f16e2e76ce0] 2: pthread_cond_wait() 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f16e24b98f0] 4: (AsyncMessenger::wait()+0x147) [0x56090bda0ab7] 5: main() 6: __libc_start_main() 7: _start() 1 jobs: ['7115778'] suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/singleton/{all/test-crash', 'supported-random-distro$/{rhel_8}}'] Failure: Command failed on smithi103 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exceptions=0\'' 1 jobs: ['7115763'] suites: ['rados/objectstore/{backends/objectstore-bluestore-a', 'supported-random-distro$/{rhel_8}}']