2022-06-16T16:30:55.672 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-16T16:30:55.675 INFO:teuthology.results:subset: '111/120000' 2022-06-16T16:30:55.675 INFO:teuthology.results:seed: '5191' 2022-06-16T21:40:04.135 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-16T21:40:04.170 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-17T00:51:22.774 INFO:teuthology.results:Tests finished! gathering results... 2022-06-17T00:51:26.660 INFO:scrape:Found 35 jobs 2022-06-17T00:51:33.083 INFO:scrape:Found 9 distinct failure reasons 2022-06-17T00:51:33.084 INFO:scrape:Failure: Cannot connect to remote host smithi061 2022-06-17T00:51:33.085 INFO:scrape:1 jobs: ['6882207'] 2022-06-17T00:51:33.086 INFO:scrape:suites: ['1-install/mimic', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/fastclose', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/rbd_cls}'] 2022-06-17T00:51:33.087 INFO:scrape: 2022-06-17T00:51:33.087 INFO:scrape:Failure: Command failed (workunit test rest/test-restful.sh) on smithi202 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.a/client.a/tmp && cd -- /home/ubuntu/cephtest/mnt.a/client.a/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=224fc22e07cebeecc3e08055cfd6105b1a30f173 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="a" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.a CEPH_ROOT=/home/ubuntu/cephtest/clone.client.a CEPH_MNT=/home/ubuntu/cephtest/mnt.a adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.a/qa/workunits/rest/test-restful.sh' 2022-06-17T00:51:33.088 INFO:scrape:1 jobs: ['6882182'] 2022-06-17T00:51:33.088 INFO:scrape:suites: ['rados/rest/{mgr-restful', 'supported-random-distro$/{ubuntu_latest}}'] 2022-06-17T00:51:33.089 INFO:scrape: 2022-06-17T00:51:33.090 INFO:scrape:Failure: Command failed on smithi025 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:224fc22e07cebeecc3e08055cfd6105b1a30f173 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 03802256-edbe-11ec-8427-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\'' 2022-06-17T00:51:33.090 INFO:scrape:1 jobs: ['6882194'] 2022-06-17T00:51:33.091 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-add}', 'rados/cephadm/osds/{0-distro/ubuntu_18.04'] 2022-06-17T00:51:33.092 INFO:scrape: 2022-06-17T00:51:33.093 INFO:scrape:Failure: [Errno 2] Cannot find file on the remote 'ubuntu@smithi130.front.sepia.ceph.com': 'rook/cluster/examples/kubernetes/ceph/operator.yaml' 2022-06-17T00:51:33.093 INFO:scrape:2 jobs: ['6882179', '6882202'] 2022-06-17T00:51:33.094 INFO:scrape:suites intersection: ['0-kubeadm', '1-rook', '3-final', 'k8s/1.21', 'net/calico', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}'] 2022-06-17T00:51:33.095 INFO:scrape:suites union: ['0-kubeadm', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}'] 2022-06-17T00:51:33.096 INFO:scrape: 2022-06-17T00:51:33.097 INFO:scrape:Failure: Command failed on smithi044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b94eae32-edbd-11ec-8427-001a4aab830c -e sha1=224fc22e07cebeecc3e08055cfd6105b1a30f173 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2022-06-17T00:51:33.098 INFO:scrape:4 jobs: ['6882196', '6882210', '6882201', '6882184'] 2022-06-17T00:51:33.098 INFO:scrape:suites intersection: ['1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools'] 2022-06-17T00:51:33.099 INFO:scrape:suites union: ['1-bootstrap/16.2.4', '1-bootstrap/octopus', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools'] 2022-06-17T00:51:33.100 INFO:scrape: 2022-06-17T00:51:33.101 INFO:scrape:Crash: Test failure: test_version (tasks.mgr.dashboard.test_api.VersionReqTest) 2022-06-17T00:51:33.101 INFO:scrape:ceph version 16.2.9-239-g224fc22e (224fc22e07cebeecc3e08055cfd6105b1a30f173) pacific (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f3b59c81ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f3b5ae55f19] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2780e2) [0x7f3b5ae560e2] 6: (DaemonServer::got_service_map()+0xb2d) [0x56431332109d] 7: (Mgr::handle_service_map(boost::intrusive_ptr)+0x1b6) [0x56431334ff76] 8: (Mgr::ms_dispatch2(boost::intrusive_ptr const&)+0x894) [0x564313352bc4] 9: (MgrStandby::ms_dispatch2(boost::intrusive_ptr const&)+0xa5) [0x56431335cb55] 10: (DispatchQueue::entry()+0x126a) [0x7f3b5b09c14a] 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3b5b14e951] 12: /lib64/libpthread.so.0(+0x81ca) [0x7f3b59c771ca] 13: clone() 2022-06-17T00:51:33.103 INFO:scrape:1 jobs: ['6882181'] 2022-06-17T00:51:33.104 INFO:scrape:suites: ['clusters/{2-node-mgr}', 'debug/mgr', 'mon_election/classic', 'rados/dashboard/{centos_8.stream_container_tools', 'random-objectstore$/{bluestore-stupid}', 'supported-random-distro$/{centos_8}', 'tasks/dashboard}'] 2022-06-17T00:51:33.104 INFO:scrape: 2022-06-17T00:51:33.105 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=224fc22e07cebeecc3e08055cfd6105b1a30f173 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2022-06-17T00:51:33.106 INFO:scrape:2 jobs: ['6882200', '6882183'] 2022-06-17T00:51:33.107 INFO:scrape:suites intersection: ['rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'task/test_e2e}'] 2022-06-17T00:51:33.108 INFO:scrape:suites union: ['rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools', 'task/test_e2e}'] 2022-06-17T00:51:33.109 INFO:scrape: 2022-06-17T00:51:33.109 INFO:scrape:Failure: 'wait for operator' reached maximum tries (90) after waiting for 900 seconds 2022-06-17T00:51:33.110 INFO:scrape:2 jobs: ['6882191', '6882211'] 2022-06-17T00:51:33.111 INFO:scrape:suites intersection: ['0-kubeadm', '1-rook', '2-workload/radosbench', '3-final', 'k8s/1.21', 'net/calico', 'rados/rook/smoke/{0-distro/ubuntu_18.04', 'rook/1.6.2}'] 2022-06-17T00:51:33.112 INFO:scrape:suites union: ['0-kubeadm', '1-rook', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'rados/rook/smoke/{0-distro/ubuntu_18.04', 'rook/1.6.2}'] 2022-06-17T00:51:33.113 INFO:scrape: 2022-06-17T00:51:33.113 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi203.front.sepia.ceph.com: ['type=AVC msg=audit(1655416700.190:18128): avc: denied { ioctl } for pid=118200 comm="iptables" path="/var/lib/containers/storage/overlay/178eb94c3500274d6338a9f1af6297d9b82ea39e954155cafc8ba11c0f5dc07a/merged" dev="overlay" ino=3803085 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1', 'type=AVC msg=audit(1655416700.292:18131): avc: denied { ioctl } for pid=118222 comm="iptables" path="/var/lib/containers/storage/overlay/178eb94c3500274d6338a9f1af6297d9b82ea39e954155cafc8ba11c0f5dc07a/merged" dev="overlay" ino=3803085 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] 2022-06-17T00:51:33.114 INFO:scrape:1 jobs: ['6882203'] 2022-06-17T00:51:33.115 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] 2022-06-17T00:51:33.116 INFO:scrape: