2023-09-19T15:51:59.044 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-19T15:51:59.045 INFO:teuthology.results:seed: '3405'
2023-09-19T16:50:37.409 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-19T16:50:37.412 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-19T20:05:51.691 INFO:teuthology.results:Tests finished! gathering results...
2023-09-19T20:05:56.965 INFO:scrape:Found 70 jobs
2023-09-19T20:05:57.402 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2023-09-19_15:51:51-rgw-wip-cbodley-testing-distro-default-smithi/7398283/teuthology.log
2023-09-19T20:05:57.403 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2023-09-19_15:51:51-rgw-wip-cbodley-testing-distro-default-smithi/7398283/teuthology.log
2023-09-19T20:05:58.728 INFO:scrape:Found 8 distinct failure reasons
2023-09-19T20:05:58.729 INFO:scrape:Failure: Command failed on smithi186 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev'
2023-09-19T20:05:58.729 INFO:scrape:1 jobs: ['7398250']
2023-09-19T20:05:58.730 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}']
2023-09-19T20:05:58.730 INFO:scrape:
2023-09-19T20:05:58.730 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fd325b2e8dd]
2023-09-19T20:05:58.731 INFO:scrape:ceph version 18.0.0-6177-g87d33202 (87d332021dd7dd1ebff94c2754207c91bcf10c5f) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd3246ca520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fd325b2e8dd]
 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fd325b2ea37]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x55b9dd69d531]
 8: radosgw-admin(+0x857b3a) [0x55b9dd6a4b3a]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55b9dd6ab19b]
 10: main()
 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fd3246b1d90]
 12: __libc_start_main()
 13: _start()
2023-09-19T20:05:58.731 INFO:scrape:1 jobs: ['7398248']
2023-09-19T20:05:58.732 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-09-19T20:05:58.732 INFO:scrape:
2023-09-19T20:05:58.732 INFO:scrape:Dead: None
2023-09-19T20:05:58.733 INFO:scrape:1 jobs: ['7398283']
2023-09-19T20:05:58.733 INFO:scrape:
2023-09-19T20:05:58.733 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f85111aa8dd]
2023-09-19T20:05:58.734 INFO:scrape:ceph version 18.0.0-6177-g87d33202 (87d332021dd7dd1ebff94c2754207c91bcf10c5f) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f850fd46520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f85111aa8dd]
 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f85111aaa37]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x55809c672531]
 8: radosgw-admin(+0x857b3a) [0x55809c679b3a]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55809c68019b]
 10: main()
 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f850fd2dd90]
 12: __libc_start_main()
 13: _start()
2023-09-19T20:05:58.734 INFO:scrape:1 jobs: ['7398245']
2023-09-19T20:05:58.735 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-09-19T20:05:58.735 INFO:scrape:
2023-09-19T20:05:58.735 INFO:scrape:Assertion: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x7f9fc34d0509]
2023-09-19T20:05:58.736 INFO:scrape:ceph version 18.0.0-6177-g87d33202 (87d332021dd7dd1ebff94c2754207c91bcf10c5f) reef (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9fc2e76420]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x7f9fc34d0509]
 5: /usr/lib/ceph/libceph-common.so.2(+0x32466b) [0x7f9fc34d066b]
 6: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x55c505f89711]
 7: radosgw-admin(+0xb9167d) [0x55c505f8267d]
 8: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x227) [0x55c505f86667]
 9: main()
 10: __libc_start_main()
 11: _start()
2023-09-19T20:05:58.736 INFO:scrape:1 jobs: ['7398278']
2023-09-19T20:05:58.736 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-09-19T20:05:58.737 INFO:scrape:
2023-09-19T20:05:58.737 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi164 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and not fails_on_dbstore'"
2023-09-19T20:05:58.738 INFO:scrape:1 jobs: ['7398230']
2023-09-19T20:05:58.738 INFO:scrape:suites: ['ignore-pg-availability', 'overrides', 'rgw/dbstore/{cluster', 's3tests-branch', 'supported-random-distro$/{centos_latest}', 'tasks/rgw_s3tests}']
2023-09-19T20:05:58.738 INFO:scrape:
2023-09-19T20:05:58.739 INFO:scrape:Failure: rgw multisite test failures
2023-09-19T20:05:58.739 INFO:scrape:2 jobs: ['7398234', '7398270']
2023-09-19T20:05:58.739 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}']
2023-09-19T20:05:58.740 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}']
2023-09-19T20:05:58.740 INFO:scrape:
2023-09-19T20:05:58.741 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fc8fdef28dd]
2023-09-19T20:05:58.741 INFO:scrape:ceph version 18.0.0-6177-g87d33202 (87d332021dd7dd1ebff94c2754207c91bcf10c5f) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc8fca8e520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fc8fdef28dd]
 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fc8fdef2a37]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x55ffb2757531]
 8: radosgw-admin(+0x857b3a) [0x55ffb275eb3a]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55ffb276519b]
 10: main()
 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc8fca75d90]
 12: __libc_start_main()
 13: _start()
2023-09-19T20:05:58.741 INFO:scrape:1 jobs: ['7398266']
2023-09-19T20:05:58.742 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-09-19T20:05:58.742 INFO:scrape: