2022-02-20T13:21:38.239 INFO:root:teuthology version: 1.1.0-eea25212
2022-02-20T13:21:38.240 INFO:teuthology.results:subset: '100/100000'
2022-02-20T13:21:38.241 INFO:teuthology.results:seed: '4589'
2022-02-20T15:54:40.896 INFO:root:teuthology version: 1.1.0-eea25212
2022-02-20T15:54:40.926 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-02-20T21:53:13.895 INFO:teuthology.results:Tests finished! gathering results...
2022-02-20T21:53:48.804 INFO:scrape:Found 309 jobs
2022-02-20T21:53:50.601 INFO:scrape:Found 9 distinct failure reasons
2022-02-20T21:53:50.602 INFO:scrape:Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)
2022-02-20T21:53:50.602 INFO:scrape:1 jobs: ['6687244']
2022-02-20T21:53:50.603 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{agent/on', 'task/test_nfs}']
2022-02-20T21:53:50.603 INFO:scrape:
2022-02-20T21:53:50.604 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
2022-02-20T21:53:50.604 INFO:scrape:2 jobs: ['6687132', '6687015']
2022-02-20T21:53:50.605 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}']
2022-02-20T21:53:50.606 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}']
2022-02-20T21:53:50.607 INFO:scrape:
2022-02-20T21:53:50.607 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh
2022-02-20T21:53:50.607 INFO:scrape:1 jobs: ['6687027']
2022-02-20T21:53:50.608 INFO:scrape:suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/force-sync-many', 'workloads/rados_api_tests}']
2022-02-20T21:53:50.608 INFO:scrape:
2022-02-20T21:53:50.609 INFO:scrape:Failure: Command failed on smithi078 with status 251: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
2022-02-20T21:53:50.609 INFO:scrape:3 jobs: ['6687238', '6687002', '6687157']
2022-02-20T21:53:50.610 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '3-final', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04']
2022-02-20T21:53:50.610 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}']
2022-02-20T21:53:50.611 INFO:scrape:
2022-02-20T21:53:50.611 INFO:scrape:Failure: {'smithi178.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'only_upgrade': False, 'package': None, 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': True, 'upgrade': None}}, 'msg': 'Failed to update apt cache: '}}
2022-02-20T21:53:50.612 INFO:scrape:1 jobs: ['6687154']
2022-02-20T21:53:50.612 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/random-eio', 'supported-random-distro$/{ubuntu_latest}}']
2022-02-20T21:53:50.613 INFO:scrape:
2022-02-20T21:53:50.613 INFO:scrape:Failure: {'smithi167.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi167.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi167.front.sepia.ceph.com port 22: No route to host\r\n', 'unreachable': True}}
2022-02-20T21:53:50.614 INFO:scrape:1 jobs: ['6687254']
2022-02-20T21:53:50.614 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}']
2022-02-20T21:53:50.615 INFO:scrape:
2022-02-20T21:53:50.615 INFO:scrape:Dead: b'2022-02-20T21:38:05.562 DEBUG:teuthology.exit:Finished running handlers'
2022-02-20T21:53:50.616 INFO:scrape:2 jobs: ['6687170', '6686980']
2022-02-20T21:53:50.616 INFO:scrape:suites intersection: ['rados']
2022-02-20T21:53:50.617 INFO:scrape:suites union: ['cluster/{12-osds', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/many', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/singleton/{all/recovery-preemption', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}']
2022-02-20T21:53:50.617 INFO:scrape:
2022-02-20T21:53:50.618 INFO:scrape:Failure: {'smithi167.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['semodule', '-i', '/tmp/nrpe.pp'], 'delta': '0:00:05.007482', 'end': '2022-02-20 15:54:02.833016', 'invocation': {'module_args': {'_raw_params': 'semodule -i /tmp/nrpe.pp', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-02-20 15:53:57.825534', 'stderr': 'libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).\nsemodule:  Failed on /tmp/nrpe.pp!', 'stderr_lines': ['libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).', 'semodule:  Failed on /tmp/nrpe.pp!'], 'stdout': '', 'stdout_lines': []}}
2022-02-20T21:53:50.618 INFO:scrape:1 jobs: ['6687255']
2022-02-20T21:53:50.619 INFO:scrape:suites: ['cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/fastclose', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}']
2022-02-20T21:53:50.620 INFO:scrape:
2022-02-20T21:53:50.620 INFO:scrape:Failure: Command failed on smithi087 with status 1: 'sudo kubeadm join 172.21.15.50:6443 --node-name smithi087 --token abcdef.s2ftqvf1w1fmqqur --discovery-token-unsafe-skip-ca-verification'
2022-02-20T21:53:50.621 INFO:scrape:1 jobs: ['6687083']
2022-02-20T21:53:50.621 INFO:scrape:suites: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '3-final', 'cluster/3-node', 'k8s/1.21', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/master}']
2022-02-20T21:53:50.623 INFO:scrape: