Found 162 jobs
Missing teuthology log /home/teuthworker/archive/mrfreezeex-2022-03-03_16:04:40-rbd-wip-mrfreezeex-multipeer-2-distro-default-smithi/6718473/teuthology.log
Found 7 distinct failure reasons
Failure: 
17 jobs: ['6718448', '6718370', '6718380', '6718399', '6718461', '6718389', '6718407', '6718453', '6718446', '6718355', '6718485', '6718501', '6718438', '6718481', '6718498', '6718507', '6718358']
suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}']
suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit-multipeer}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']

Dead: None
1 jobs: ['6718473']

Failure: reached maximum tries (90) after waiting for 540 seconds
4 jobs: ['6718480', '6718430', '6718479', '6718492']
suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}', 'supported-random-distro$/{ubuntu_latest}']
suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-journal-stress-workunit}', 'workloads/rbd-mirror-snapshot-bootstrap-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}']

Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi135 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=090c8419c748ee700b7ab0cdc033c1586ff3fa0d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'
1 jobs: ['6718414']
suites: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-minimum}']

Failure: "2022-03-03T20:28:37.276848+0000 mon.a (mon.0) 95 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log
7 jobs: ['6718363', '6718432', '6718422', '6718404', '6718478', '6718376', '6718401']
suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}']
suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-ha-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff}', 'workloads/rbd-mirror-workunit-policy-none}']

Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi166 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=090c8419c748ee700b7ab0cdc033c1586ff3fa0d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'
2 jobs: ['6718447', '6718382']
suites intersection: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'openstack}', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit}']
suites union: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit}']

Failure: {'smithi008.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'only_upgrade': False, 'package': None, 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': True, 'upgrade': None}}, 'msg': 'Failed to update apt cache: '}}
1 jobs: ['6718474']
suites: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-workunit-policy-none}']