2024-03-10T04:29:27.123 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-10T04:29:27.123 INFO:teuthology.results:seed: '2141' 2024-03-10T05:10:11.495 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-10T05:10:11.521 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-10T09:50:34.377 INFO:teuthology.results:Tests finished! gathering results... 2024-03-10T09:50:39.805 INFO:scrape:Found 73 jobs 2024-03-10T09:50:45.994 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2024-03-09_13:49:07-rgw-wip-cbodley-testing-distro-default-smithi/7588537/teuthology.log 2024-03-10T09:50:45.995 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2024-03-09_13:49:07-rgw-wip-cbodley-testing-distro-default-smithi/7588537/teuthology.log 2024-03-10T09:50:45.995 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2024-03-09_13:49:07-rgw-wip-cbodley-testing-distro-default-smithi/7588537/teuthology.log 2024-03-10T09:50:45.995 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2024-03-09_13:49:07-rgw-wip-cbodley-testing-distro-default-smithi/7588537/teuthology.log 2024-03-10T09:50:45.995 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2024-03-09_13:49:07-rgw-wip-cbodley-testing-distro-default-smithi/7588537/teuthology.log 2024-03-10T09:50:45.995 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2024-03-09_13:49:07-rgw-wip-cbodley-testing-distro-default-smithi/7588537/teuthology.log 2024-03-10T09:50:45.995 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2024-03-09_13:49:07-rgw-wip-cbodley-testing-distro-default-smithi/7588537/teuthology.log 2024-03-10T09:50:45.996 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2024-03-09_13:49:07-rgw-wip-cbodley-testing-distro-default-smithi/7588537/teuthology.log 2024-03-10T09:50:45.996 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2024-03-09_13:49:07-rgw-wip-cbodley-testing-distro-default-smithi/7588537/teuthology.log 2024-03-10T09:50:54.594 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.595 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:54.638 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.638 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:54.680 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.680 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:54.722 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.722 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:54.763 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.763 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:54.803 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.803 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:54.843 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.843 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:54.883 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.883 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:54.925 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.925 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:54.968 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:54.968 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:55.008 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:55.009 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:55.049 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:55.050 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:55.089 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:55.089 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:55.130 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:55.130 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:55.170 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:55.171 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:55.218 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:55.218 WARNING:scrape:Not-understood crash indication 2024-03-10T09:50:55.260 WARNING:scrape:Not-understood crash indication 2024-03-10T05:20:29.818 INFO:tasks.rgw.client.0.smithi032.stderr:daemon-helper: command crashed with signal 6 2024-03-10T09:50:55.261 WARNING:scrape:Not-understood crash indication 2024-03-10T09:51:11.610 INFO:scrape:Found 27 distinct failure reasons 2024-03-10T09:51:11.611 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi102 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" 2024-03-10T09:51:11.611 INFO:scrape:14 jobs: ['7588550', '7588569', '7588527', '7588513', '7588566', '7588519', '7588502', '7588553', '7588504', '7588539', '7588514', '7588547', '7588506', '7588526'] 2024-03-10T09:51:11.611 INFO:scrape:suites intersection: ['ignore-pg-availability', 's3tests-branch'] 2024-03-10T09:51:11.611 INFO:scrape:suites union: ['1-ceph-install/install', '1-keycloak', '2-kms/barbican', '2-kms/testing', '2-kms/vault_kv', '2-kms/vault_old', '2-kms/vault_transit', '2-s3tests}}', '3-rgw/rgw', '4-tests/{s3tests}', 'clusters/fixed-1', 'frontend/beast', 'http', 'ignore-pg-availability', 'install', 'objectstore', 'objectstore/bluestore-bitmap', 'overrides', 'pool-type', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/dbstore/{cluster', 'rgw/lifecycle/{cluster', 'rgw/multifs/{clusters/fixed-2', 'rgw/sts/{cluster', 'rgw/tempest/{0-install', 'rgw/thrash/{clusters/fixed-2', 'rgw/website/{clusters/fixed-2', 'rgw_frontend/beast', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-branch', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rgw_s3tests', 'tasks/rgw_s3tests}', 'tasks/s3tests', 'tasks/s3tests-website', 'tasks/{0-install', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'ubuntu_latest}', 'workload/rgw_s3tests}'] 2024-03-10T09:51:11.612 INFO:scrape: 2024-03-10T09:51:11.612 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f62e3580ee0] 2024-03-10T09:51:11.612 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f62e2454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f62e24a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f62e3580ee0] 6: /usr/lib64/ceph/libceph-common.so.2(+0x181044) [0x7f62e3581044] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557ed7222c71] 8: radosgw-admin(+0x7e5a6a) [0x557ed722aa6a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x557ed722f27b] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7f62e243feb0] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.612 INFO:scrape:1 jobs: ['7588541'] 2024-03-10T09:51:11.612 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.612 INFO:scrape: 2024-03-10T09:51:11.612 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6eccf0fdd5] 2024-03-10T09:51:11.612 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6ecbd93520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6eccf0fdd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f6eccf0ff2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x560b69a1a801] 8: radosgw-admin(+0x8bd21a) [0x560b69a2621a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x560b69a2a87b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6ecbd7ad90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.612 INFO:scrape:1 jobs: ['7588565'] 2024-03-10T09:51:11.613 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.613 INFO:scrape: 2024-03-10T09:51:11.613 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f7fcc294dd5] 2024-03-10T09:51:11.613 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7fcb118520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f7fcc294dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7fcc294f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557b1edaf801] 8: radosgw-admin(+0x8bd21a) [0x557b1edbb21a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x557b1edbf87b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f7fcb0ffd90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.613 INFO:scrape:1 jobs: ['7588533'] 2024-03-10T09:51:11.613 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.613 INFO:scrape: 2024-03-10T09:51:11.613 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f96b0b80ee0] 2024-03-10T09:51:11.613 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f96afa54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f96afaa154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f96b0b80ee0] 6: /usr/lib64/ceph/libceph-common.so.2(+0x181044) [0x7f96b0b81044] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x563474e8dc71] 8: radosgw-admin(+0x7e5a6a) [0x563474e95a6a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x563474e9a27b] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7f96afa3feb0] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.613 INFO:scrape:1 jobs: ['7588542'] 2024-03-10T09:51:11.613 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.614 INFO:scrape: 2024-03-10T09:51:11.614 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fdb6d580ee0] 2024-03-10T09:51:11.614 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fdb6c454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fdb6c4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fdb6d580ee0] 6: /usr/lib64/ceph/libceph-common.so.2(+0x181044) [0x7fdb6d581044] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d8edf73c71] 8: radosgw-admin(+0x7e5a6a) [0x55d8edf7ba6a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55d8edf8027b] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7fdb6c43feb0] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.614 INFO:scrape:1 jobs: ['7588520'] 2024-03-10T09:51:11.614 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.614 INFO:scrape: 2024-03-10T09:51:11.614 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fc484e40dd5] 2024-03-10T09:51:11.614 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc483cc4520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fc484e40dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7fc484e40f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x556b7f90e801] 8: radosgw-admin(+0x8bd21a) [0x556b7f91a21a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x556b7f91e87b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc483cabd90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.614 INFO:scrape:1 jobs: ['7588525'] 2024-03-10T09:51:11.614 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.615 INFO:scrape: 2024-03-10T09:51:11.615 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f4c4903cdd5] 2024-03-10T09:51:11.615 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4c47ec0520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f4c4903cdd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f4c4903cf2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55692cb03801] 8: radosgw-admin(+0x8bd21a) [0x55692cb0f21a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55692cb1387b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f4c47ea7d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.615 INFO:scrape:1 jobs: ['7588518'] 2024-03-10T09:51:11.615 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.615 INFO:scrape: 2024-03-10T09:51:11.615 INFO:scrape:Failure: Command failed (workunit test rgw/run-bucket-check.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b792cf723d310879ca5186de8cfccd8a89ae719a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh' 2024-03-10T09:51:11.615 INFO:scrape:1 jobs: ['7588528'] 2024-03-10T09:51:11.615 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.615 INFO:scrape: 2024-03-10T09:51:11.616 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3809fcbdd5] 2024-03-10T09:51:11.616 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3808e4f520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3809fcbdd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f3809fcbf2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562429baf801] 8: radosgw-admin(+0x8bd21a) [0x562429bbb21a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x562429bbf87b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f3808e36d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.616 INFO:scrape:1 jobs: ['7588538'] 2024-03-10T09:51:11.616 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.616 INFO:scrape: 2024-03-10T09:51:11.616 INFO:scrape:Dead: None 2024-03-10T09:51:11.616 INFO:scrape:1 jobs: ['7588537'] 2024-03-10T09:51:11.616 INFO:scrape: 2024-03-10T09:51:11.616 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f8be2780ee0] 2024-03-10T09:51:11.616 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8be1654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8be16a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f8be2780ee0] 6: /usr/lib64/ceph/libceph-common.so.2(+0x181044) [0x7f8be2781044] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55ede6bb9c71] 8: radosgw-admin(+0x7e5a6a) [0x55ede6bc1a6a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55ede6bc627b] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7f8be163feb0] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.616 INFO:scrape:1 jobs: ['7588568'] 2024-03-10T09:51:11.616 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.616 INFO:scrape: 2024-03-10T09:51:11.617 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fa934abcdd5] 2024-03-10T09:51:11.617 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fa933940520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fa934abcdd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7fa934abcf2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55731b415801] 8: radosgw-admin(+0x8bd21a) [0x55731b42121a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55731b42587b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fa933927d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.617 INFO:scrape:1 jobs: ['7588555'] 2024-03-10T09:51:11.617 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.617 INFO:scrape: 2024-03-10T09:51:11.617 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f34f1f80ee0] 2024-03-10T09:51:11.617 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f34f0e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f34f0ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f34f1f80ee0] 6: /usr/lib64/ceph/libceph-common.so.2(+0x181044) [0x7f34f1f81044] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a9b9f0bc71] 8: radosgw-admin(+0x7e5a6a) [0x55a9b9f13a6a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a9b9f1827b] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7f34f0e3feb0] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.617 INFO:scrape:1 jobs: ['7588549'] 2024-03-10T09:51:11.617 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.617 INFO:scrape: 2024-03-10T09:51:11.617 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f1607156dd5] 2024-03-10T09:51:11.618 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1605fda520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f1607156dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f1607156f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562971513801] 8: radosgw-admin(+0x8bd21a) [0x56297151f21a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x56297152387b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f1605fc1d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.618 INFO:scrape:1 jobs: ['7588559'] 2024-03-10T09:51:11.618 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.618 INFO:scrape: 2024-03-10T09:51:11.618 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fad015d5dd5] 2024-03-10T09:51:11.618 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fad00459520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fad015d5dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7fad015d5f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5573ab588801] 8: radosgw-admin(+0x8bd21a) [0x5573ab59421a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5573ab59887b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fad00440d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.618 INFO:scrape:1 jobs: ['7588571'] 2024-03-10T09:51:11.618 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.618 INFO:scrape: 2024-03-10T09:51:11.618 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fac3ca49dd5] 2024-03-10T09:51:11.618 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fac3b8cd520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fac3ca49dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7fac3ca49f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x564b99ba2801] 8: radosgw-admin(+0x8bd21a) [0x564b99bae21a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x564b99bb287b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fac3b8b4d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.619 INFO:scrape:1 jobs: ['7588523'] 2024-03-10T09:51:11.619 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.619 INFO:scrape: 2024-03-10T09:51:11.619 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi046 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a amqp_test' 2024-03-10T09:51:11.619 INFO:scrape:2 jobs: ['7588510', '7588535'] 2024-03-10T09:51:11.619 INFO:scrape:suites intersection: ['bluestore-bitmap', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast'] 2024-03-10T09:51:11.619 INFO:scrape:suites union: ['bluestore-bitmap', 'centos_latest', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'supported-distros/{centos_latest}', 'tasks/amqp/{0-install', 'tasks/kafka/{0-install', 'test_amqp}}', 'test_kafka}}'] 2024-03-10T09:51:11.619 INFO:scrape: 2024-03-10T09:51:11.619 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fe6be855dd5] 2024-03-10T09:51:11.619 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fe6bd6d9520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fe6be855dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7fe6be855f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5649c73cb801] 8: radosgw-admin(+0x8bd21a) [0x5649c73d721a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5649c73db87b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fe6bd6c0d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.619 INFO:scrape:1 jobs: ['7588570'] 2024-03-10T09:51:11.620 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.620 INFO:scrape: 2024-03-10T09:51:11.620 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ffbf909edd5] 2024-03-10T09:51:11.620 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ffbf7f22520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ffbf909edd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7ffbf909ef2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x563a43c22801] 8: radosgw-admin(+0x8bd21a) [0x563a43c2e21a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x563a43c3287b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ffbf7f09d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.620 INFO:scrape:1 jobs: ['7588551'] 2024-03-10T09:51:11.620 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.620 INFO:scrape: 2024-03-10T09:51:11.620 INFO:scrape:Failure: rgw multisite test failures 2024-03-10T09:51:11.620 INFO:scrape:2 jobs: ['7588509', '7588545'] 2024-03-10T09:51:11.620 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2024-03-10T09:51:11.621 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2024-03-10T09:51:11.621 INFO:scrape: 2024-03-10T09:51:11.621 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f5c21e57dd5] 2024-03-10T09:51:11.621 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5c20cdb520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f5c21e57dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f5c21e57f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55bb6c4d5801] 8: radosgw-admin(+0x8bd21a) [0x55bb6c4e121a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55bb6c4e587b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f5c20cc2d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.621 INFO:scrape:1 jobs: ['7588564'] 2024-03-10T09:51:11.621 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.621 INFO:scrape: 2024-03-10T09:51:11.621 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f617f8d0dd5] 2024-03-10T09:51:11.621 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f617e754520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f617f8d0dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f617f8d0f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5562dc855801] 8: radosgw-admin(+0x8bd21a) [0x5562dc86121a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5562dc86587b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f617e73bd90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.621 INFO:scrape:1 jobs: ['7588560'] 2024-03-10T09:51:11.621 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.622 INFO:scrape: 2024-03-10T09:51:11.622 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2d6bfd0dd5] 2024-03-10T09:51:11.622 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2d6ae54520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2d6bfd0dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f2d6bfd0f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5617fd71e801] 8: radosgw-admin(+0x8bd21a) [0x5617fd72a21a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5617fd72e87b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f2d6ae3bd90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.622 INFO:scrape:1 jobs: ['7588573'] 2024-03-10T09:51:11.622 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-10T09:51:11.622 INFO:scrape: 2024-03-10T09:51:11.622 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2fd5547dd5] 2024-03-10T09:51:11.622 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2fd43cb520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2fd5547dd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f2fd5547f2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55f04e5ee801] 8: radosgw-admin(+0x8bd21a) [0x55f04e5fa21a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55f04e5fe87b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f2fd43b2d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.622 INFO:scrape:1 jobs: ['7588522'] 2024-03-10T09:51:11.622 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.623 INFO:scrape: 2024-03-10T09:51:11.623 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f7f524dadd5] 2024-03-10T09:51:11.623 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7f5135e520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f7f524dadd5] 6: /usr/lib/ceph/libceph-common.so.2(+0x180f2f) [0x7f7f524daf2f] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x555b35aba801] 8: radosgw-admin(+0x8bd21a) [0x555b35ac621a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x555b35aca87b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f7f51345d90] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.623 INFO:scrape:1 jobs: ['7588552'] 2024-03-10T09:51:11.623 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.623 INFO:scrape: 2024-03-10T09:51:11.623 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7ff95e980ee0] 2024-03-10T09:51:11.623 INFO:scrape:ceph version 19.0.0-1976-gb792cf72 (b792cf723d310879ca5186de8cfccd8a89ae719a) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7ff95d854db0] 2: /lib64/libc.so.6(+0xa154c) [0x7ff95d8a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7ff95e980ee0] 6: /usr/lib64/ceph/libceph-common.so.2(+0x181044) [0x7ff95e981044] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5596c1546c71] 8: radosgw-admin(+0x7e5a6a) [0x5596c154ea6a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5596c155327b] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7ff95d83feb0] 12: __libc_start_main() 13: _start() 2024-03-10T09:51:11.623 INFO:scrape:1 jobs: ['7588531'] 2024-03-10T09:51:11.623 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-10T09:51:11.623 INFO:scrape: