Found 195 jobs Found 6 distinct failure reasons Dead: b'2022-08-19T01:40:59.412 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['6976853'] suites: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] Failure: "2022-08-17T18:13:56.202189+0000 mon.a (mon.0) 105 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 14 jobs: ['6976765', '6976702', '6976808', '6976777', '6976674', '6976767', '6976843', '6976842', '6976741', '6976718', '6976852', '6976859', '6976776', '6976720'] suites intersection: ['mirror', 'msgr-failures/few', 'roles}'] suites union: ['cluster/2-nodes/{base/{install', 'cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-journal-workunit-multipeer}', 'workloads/rbd-mirror-journal-workunit}', 'workloads/rbd-mirror-snapshot-bootstrap-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff}'] Failure: {'smithi101.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': 'ceph/python3-asyncssh', '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': ['dnf', '-y', 'copr', 'enable', 'ceph/python3-asyncssh'], 'delta': '0:02:09.131683', 'end': '2022-08-18 12:37:54.725807', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'dnf -y copr enable ceph/python3-asyncssh', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': 'ceph/python3-asyncssh', 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-08-18 12:35:45.594124', 'stderr': 'Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out', 'stderr_lines': ['Enabling a Copr repository. Please note that this repository is not part', 'of the main distribution, and quality may vary.', '', 'The Fedora Project does not exercise any power over the contents of', 'this repository beyond the rules outlined in the Copr FAQ at', ',', 'and packages are not held to any quality or security level.', '', 'Please do not file bug reports about these packages in Fedora', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out'], 'stdout': '', 'stdout_lines': []}], 'warnings': ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]}, 'smithi007.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': 'ceph/python3-asyncssh', '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': ['dnf', '-y', 'copr', 'enable', 'ceph/python3-asyncssh'], 'delta': '0:02:09.756497', 'end': '2022-08-18 12:37:59.431717', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'dnf -y copr enable ceph/python3-asyncssh', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': 'ceph/python3-asyncssh', 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-08-18 12:35:49.675220', 'stderr': 'Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out', 'stderr_lines': ['Enabling a Copr repository. Please note that this repository is not part', 'of the main distribution, and quality may vary.', '', 'The Fedora Project does not exercise any power over the contents of', 'this repository beyond the rules outlined in the Copr FAQ at', ',', 'and packages are not held to any quality or security level.', '', 'Please do not file bug reports about these packages in Fedora', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out'], 'stdout': '', 'stdout_lines': []}], 'warnings': ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]}} 1 jobs: ['6976801'] suites: ['cluster/2-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}'] Failure: reached maximum tries (50) after waiting for 300 seconds 2 jobs: ['6976711', '6976815'] suites intersection: ['extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}'] suites union: ['cluster/2-nodes/{base/{install', 'cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}'] Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi201 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5617484c272c369a8e3eaacbe1679b000923297f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 1 jobs: ['6976752'] suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] Failure: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi097 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5617484c272c369a8e3eaacbe1679b000923297f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=journal MIRROR_POOL_MODE=pool RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 1 jobs: ['6976855'] suites: ['cluster/2-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-stress-workunit}']