Found 64 jobs Found 9 distinct failure reasons Failure: failed to recover before timeout expired 1 jobs: ['5670806'] suites: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_16.04', 'thrashosds-health}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack'] Dead: b'2020-12-01T15:34:42.746 DEBUG:teuthology.exit:Finished running handlers' 12 jobs: ['5670853', '5670833', '5670865', '5670845', '5670798', '5670821', '5670825', '5670790', '5670841', '5670854', '5670842', '5670861'] suites intersection: ['2-partial-upgrade/firsthalf', '3-thrash/default', '5-finish-upgrade', 'start}', 'thrashosds-health}'] suites union: ['1-ceph-install/mimic', '1-luminous-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-ec-workload', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload', '7-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/centos_latest', 'supported-all-distro/rhel_7', 'supported-all-distro/ubuntu_16.04', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack'] Failure: psutil.NoSuchProcess process no longer exists (pid=29759) 2 jobs: ['5670814', '5670857'] suites intersection: ['1-ceph-install/mimic', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload/{rbd-python', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack'] suites union: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/rhel_7', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack'] Dead: b'2020-12-01T15:28:40.441 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 0)...' 2 jobs: ['5670849', '5670846'] Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=mimic TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' 1 jobs: ['5670836'] suites: ['1-ceph-install/mimic', '1.1-pg-log-overrides/short_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-all', '4-msgr2', '4-nautilus', '5-final-workload/{blogbench', 'ec-rados-default', 'objectstore/bluestore-bitmap', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check', 'rgw_ragweed_prepare', 'rgw_swift}', 'start}', 'supported-all-distro/centos_latest}', 'test_rbd_api', 'test_rbd_python}', 'upgrade:mimic-x/parallel/{0-cluster/{openstack'] Failure: SELinux denials found on ubuntu@smithi005.front.sepia.ceph.com: ['type=AVC msg=audit(1606798201.744:4756): avc: denied { getattr } for pid=36859 comm="sadc" path="/proc/36859/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606801831.192:7253): avc: denied { getattr } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606800001.271:5515): avc: denied { read } for pid=43431 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606800601.308:5596): avc: denied { getattr } for pid=44194 comm="sadc" path="/proc/44194/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606798231.210:4760): avc: denied { getattr } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606797031.193:4737): avc: denied { read } for pid=18105 comm="ntpd" name="if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606801831.192:7252): avc: denied { read } for pid=18105 comm="ntpd" name="if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606794001.473:4585): avc: denied { open } for pid=30964 comm="sadc" path="/proc/30964/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606800601.308:5595): avc: denied { open } for pid=44194 comm="sadc" path="/proc/44194/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606798201.744:4755): avc: denied { open } for pid=36859 comm="sadc" path="/proc/36859/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606800331.174:5566): avc: denied { getattr } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606794001.473:4585): avc: denied { read } for pid=30964 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606799731.200:5506): avc: denied { read } for pid=18105 comm="ntpd" name="if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606800331.174:5565): avc: denied { read } for pid=18105 comm="ntpd" name="if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606800001.271:5515): avc: denied { open } for pid=43431 comm="sadc" path="/proc/43431/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606795201.538:4679): avc: denied { read } for pid=32499 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606800331.174:5565): avc: denied { open } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606801831.192:7252): avc: denied { open } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606802401.399:8782): avc: denied { read } for pid=68560 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606800001.271:5516): avc: denied { getattr } for pid=43431 comm="sadc" path="/proc/43431/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606800601.308:5595): avc: denied { read } for pid=44194 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606797601.710:4744): avc: denied { read } for pid=34643 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606802401.399:8783): avc: denied { getattr } for pid=68560 comm="sadc" path="/proc/68560/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606793731.181:4579): avc: denied { getattr } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606797031.193:4738): avc: denied { getattr } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606798231.210:4759): avc: denied { read } for pid=18105 comm="ntpd" name="if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606797031.193:4737): avc: denied { open } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606793731.180:4578): avc: denied { read } for pid=18105 comm="ntpd" name="if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606794001.473:4586): avc: denied { getattr } for pid=30964 comm="sadc" path="/proc/30964/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606803031.206:10129): avc: denied { getattr } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606794631.222:4625): avc: denied { getattr } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606798201.744:4755): avc: denied { read } for pid=36859 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606794631.222:4624): avc: denied { open } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606803031.206:10128): avc: denied { open } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606797601.711:4745): avc: denied { getattr } for pid=34643 comm="sadc" path="/proc/34643/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606799731.200:5506): avc: denied { open } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606797601.710:4744): avc: denied { open } for pid=34643 comm="sadc" path="/proc/34643/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606798231.210:4759): avc: denied { open } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606803031.206:10128): avc: denied { read } for pid=18105 comm="ntpd" name="if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606799731.201:5507): avc: denied { getattr } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606793731.180:4578): avc: denied { open } for pid=18105 comm="ntpd" path="/proc/18105/net/if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606795201.539:4680): avc: denied { getattr } for pid=32499 comm="sadc" path="/proc/32499/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606794631.222:4624): avc: denied { read } for pid=18105 comm="ntpd" name="if_inet6" dev="proc" ino=4026532227 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606802401.399:8782): avc: denied { open } for pid=68560 comm="sadc" path="/proc/68560/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1606795201.538:4679): avc: denied { open } for pid=32499 comm="sadc" path="/proc/32499/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1'] 1 jobs: ['5670823'] suites: ['1-ceph-install/mimic', '1.1-pg-log-overrides/short_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-all', '4-msgr2', '4-nautilus', '5-final-workload/{blogbench', 'ec-rados-default', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check', 'rgw_ragweed_prepare', 'rgw_swift}', 'start}', 'supported-all-distro/rhel_7}', 'test_rbd_api', 'test_rbd_python}', 'upgrade:mimic-x/parallel/{0-cluster/{openstack'] Failure: Command failed on smithi134 with status 100: 'sudo apt-get clean' 1 jobs: ['5670838'] suites: ['1-luminous-install/mimic', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-ec-workload', '5-finish-upgrade', '7-final-workload', 'objectstore/bluestore-bitmap', 'start}', 'supported-all-distro/centos_latest', 'thrashosds-health}', 'upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack'] Failure: SSH connection to smithi060 was lost: 'uname -r' 1 jobs: ['5670837'] suites: ['1-ceph-install/mimic', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/centos_latest', 'thrashosds-health}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack'] Dead: b'psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=18297)' 1 jobs: ['5670829'] suites: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_16.04', 'thrashosds-health}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']