Found 70 jobs Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:10.907 INFO:tasks.rgw.client.0.smithi031.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2023-12-04T19:12:00.568 INFO:tasks.rgw.c1.client.0.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T19:12:06.589 INFO:tasks.rgw.c1.client.1.smithi151.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.217 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-12-04T20:50:32.223 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi151.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Found 16 distinct failure reasons Failure: ['Failed to update apt cache: unknown reason'] 7 jobs: ['7477247', '7477290', '7477270', '7477283', '7477300', '7477273', '7477248'] suites intersection: ['ignore-pg-availability'] suites union: ['1-ceph-install/install', '1-keycloak', '2-kms/vault_transit', '2-s3tests}}', '3-rgw/rgw', '4-tests/{s3tests}', 'cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore', 'objectstore/bluestore-bitmap', 'overrides', 'pool-type', 'proto/http', 'ragweed', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/multifs/{clusters/fixed-2', 'rgw/singleton/{all/radosgw-admin', 'rgw/sts/{cluster', 'rgw/verify/{0-install', 'rgw_frontend/beast', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/rgw_bucket_quota', 'tasks/{0-install', 'tasks/{bucket-check', 'ubuntu_latest}', 'validater/lockdep}', 'validater/valgrind}', 'versioning}'] Failure: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2 jobs: ['7477252', '7477296'] suites intersection: ['2-setup', 'cluster', 'frontend/beast', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'overrides}', 'rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream}'] suites union: ['2-setup', '3-upgrade-sequence/osds-then-rgws', '3-upgrade-sequence/rgws-then-osds', 'cluster', 'frontend/beast', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'overrides}', 'rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f60cd903ef0] ceph version 18.0.0-7617-g8c6118ee (8c6118eee0f4440cb3687f9c8e2b718fd2d59013) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f60cc6b7520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f60cd903ef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f60cd90404a] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x564650318911] 8: radosgw-admin(+0x85ba5a) [0x564650321a5a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x56465032609b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f60cc69ed90] 12: __libc_start_main() 13: _start() 1 jobs: ['7477301'] suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] Failure: ['CPAN command failed: /usr/bin/cpanm Amazon::S3'] 21 jobs: ['7477285', '7477292', '7477303', '7477266', '7477249', '7477293', '7477304', '7477268', '7477277', '7477267', '7477278', '7477246', '7477305', '7477291', '7477256', '7477281', '7477287', '7477294', '7477263', '7477284', '7477259'] suites intersection: ['ignore-pg-availability'] suites union: ['1-ceph-install/install', '2-kms/vault_kv', '3-rgw/rgw', '4-tests/{s3tests}', 'cls', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'install', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/multifs/{clusters/fixed-2', 'rgw/service-token/{clusters/fixed-1', 'rgw/singleton/{all/radosgw-admin', 'rgw/tempest/{clusters/fixed-1', 'rgw/thrash/{clusters/fixed-2', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/rgw_multipart_upload', 'tasks/rgw_ragweed', 'tasks/rgw_s3tests', 'tasks/rgw_tempest', 'tasks/rgw_user_quota', 'tasks/service-token', 'tasks/{bucket-check', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'ubuntu_latest}', 'validater/lockdep}', 'validater/valgrind}', 'versioning}', 'workload/rgw_s3tests}'] Failure: Command failed on smithi042 with status 1: 'cd /home/ubuntu/cephtest/hadoop/hadoop-tools/hadoop-aws/ && rm src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java && /home/ubuntu/cephtest/apache-maven-3.6.3/bin/mvn test -Dtest=S3a*,TestS3A*' 1 jobs: ['7477241'] suites: ['hadoop/default', 'ignore-pg-availability', 'overrides', 'rgw/hadoop-s3a/{clusters/fixed-2', 's3a-hadoop}'] Failure: ['Connection timed out', 'Unable to connect to archive.ubuntu.com:http:', 'connection timed out'] 12 jobs: ['7477258', '7477302', '7477280', '7477254', '7477265', '7477272', '7477286', '7477297', '7477295', '7477276', '7477274', '7477264'] suites intersection: ['frontend/beast', 'ignore-pg-availability'] suites union: ['2-setup', '3-upgrade-sequence/rgws-then-osds', 'cls', 'cluster', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'http', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'install', 'mp_reupload', 'msgr-failures/few', 'notify', 'objectstore/bluestore-bitmap', 'omap_limits', 'overrides', 'overrides}', 'proto/http', 'proto/https', 'ragweed', 'realms/two-zonegroup', 'reshard', 'rgw/multifs/{clusters/fixed-2', 'rgw/multisite/{clusters', 'rgw/thrash/{clusters/fixed-2', 'rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04}', 'rgw/verify/{0-install', 'rgw/website/{clusters/fixed-2', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rgw_multipart_upload', 'tasks/rgw_s3tests', 'tasks/s3tests-website', 'tasks/test_multi}', 'tasks/{bucket-check', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'ubuntu_latest}', 'validater/lockdep}', 'validater/valgrind}', 'versioning}', 'workload/rgw_multipart_upload}', 'workload/rgw_user_quota}'] Failure: ['connection timed out'] 5 jobs: ['7477279', '7477288', '7477298', '7477242', '7477275'] suites intersection: ['ignore-pg-availability', 'overrides'] suites union: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'hadoop/v32', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/hadoop-s3a/{clusters/fixed-2', 'rgw/lifecycle/{cluster', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 's3a-hadoop}', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rgw_s3tests}', 'tasks/{bucket-check', 'validater/lockdep}', 'validater/valgrind}', 'versioning}'] Failure: smithi029.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - dnf-utils - sysstat - libedit - boost-thread - xfsprogs - gdisk - parted - libgcrypt - fuse-libs - openssl - libuuid - podman - attr - lsof - gettext - bc - xfsdump - blktrace - usbredir - valgrind - nfs-utils - ncurses-devel - gcc - git - genisoimage - qemu-img - qemu-kvm-core - qemu-kvm-block-rbd - libacl-devel - autoconf - gdb - iozone releasever: null security: false skip_broken: false state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download packages: dnf-plugins-core-4.3.0-12.el9.noarch: Cannot download, all mirrors were already tried without success' results: []smithi012.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - dnf-utils - sysstat - libedit - boost-thread - xfsprogs - gdisk - parted - libgcrypt - fuse-libs - openssl - libuuid - podman - attr - lsof - gettext - bc - xfsdump - blktrace - usbredir - valgrind - nfs-utils - ncurses-devel - gcc - git - genisoimage - qemu-img - qemu-kvm-core - qemu-kvm-block-rbd - libacl-devel - autoconf - gdb - iozone releasever: null security: false skip_broken: false state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download packages: yum-utils-4.3.0-12.el9.noarch: Cannot download, all mirrors were already tried without success' results: [] 1 jobs: ['7477282'] suites: ['2-setup', '3-upgrade-sequence/osds-then-rgws', 'cluster', 'frontend/beast', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'overrides}', 'rgw/upgrade/{1-install/quincy/{distro$/{centos_latest}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f0424b96ef0] ceph version 18.0.0-7617-g8c6118ee (8c6118eee0f4440cb3687f9c8e2b718fd2d59013) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f042394a520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f0424b96ef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f0424b9704a] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x561889f79911] 8: radosgw-admin(+0x85ba5a) [0x561889f82a5a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x561889f8709b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f0423931d90] 12: __libc_start_main() 13: _start() 1 jobs: ['7477257'] suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] Dead: b'2023-12-04T19:15:16.178 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml' 1 jobs: ['7477306'] suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] Failure: Command failed on smithi028 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev' 2 jobs: ['7477260', '7477261'] suites intersection: ['ignore-pg-availability', 's3tests-branch'] suites union: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'ubuntu_latest}', 'validater/lockdep}', 'versioning}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f5e5bd70ef0] ceph version 18.0.0-7617-g8c6118ee (8c6118eee0f4440cb3687f9c8e2b718fd2d59013) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5e5ab24520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f5e5bd70ef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f5e5bd7104a] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55cf2a297911] 8: radosgw-admin(+0x85ba5a) [0x55cf2a2a0a5a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55cf2a2a509b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f5e5ab0bd90] 12: __libc_start_main() 13: _start() 1 jobs: ['7477255'] suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] Failure: [Errno 13] Permission denied: '/vstart_runner.log' 1 jobs: ['7477271'] suites: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/singleton/{all/radosgw-admin', 'rgw_pool_type/ec', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (bucket notification tests against different endpoints) on smithi031 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test' 1 jobs: ['7477245'] suites: ['bluestore-bitmap', 'centos_latest', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}'] Failure: rgw multisite test failures 1 jobs: ['7477244'] suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f195caa0ef0] ceph version 18.0.0-7617-g8c6118ee (8c6118eee0f4440cb3687f9c8e2b718fd2d59013) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f195b854520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f195caa0ef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f195caa104a] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562ba506e911] 8: radosgw-admin(+0x85ba5a) [0x562ba5077a5a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x562ba507c09b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f195b83bd90] 12: __libc_start_main() 13: _start() 1 jobs: ['7477253'] suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}']