2023-12-14T20:17:57.130 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e
2023-12-14T20:17:57.130 INFO:teuthology.results:seed: '323'
2023-12-14T20:19:01.663 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e
2023-12-14T20:19:01.669 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-12-15T00:04:18.580 INFO:teuthology.results:Tests finished! gathering results...
2023-12-15T00:04:20.612 INFO:scrape:Found 23 jobs
2023-12-15T00:04:22.286 INFO:scrape:Found 4 distinct failure reasons
2023-12-15T00:04:22.287 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi019 with status 1: "BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a '!kafka_test,!amqp_test,!amqp_ssl_test,!kafka_security_test,!modification_required,!manual_test'"
2023-12-15T00:04:22.287 INFO:scrape:1 jobs: ['7491462']
2023-12-15T00:04:22.287 INFO:scrape:suites: ['bluestore-bitmap', 'centos_latest', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}']
2023-12-15T00:04:22.287 INFO:scrape:
2023-12-15T00:04:22.287 INFO:scrape:Failure: rgw multisite test failures
2023-12-15T00:04:22.287 INFO:scrape:2 jobs: ['7491461', '7491471']
2023-12-15T00:04:22.287 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-12-15T00:04:22.288 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-12-15T00:04:22.288 INFO:scrape:
2023-12-15T00:04:22.288 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fc95cb6ff51]
2023-12-15T00:04:22.288 INFO:scrape:ceph version 19.0.0-176-g4a1cf0b4 (4a1cf0b45eccd2f4955bf9c26d3227564def63e5) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7fc95a454db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7fc95a4a154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fc95cb6ff51]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x1700b5) [0x7fc95cb700b5]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x5649834a8541]
 8: radosgw-admin(+0x78b332) [0x5649834b6332]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5649834bf57b]
 10: main()
 11: /lib64/libc.so.6(+0x3feb0) [0x7fc95a43feb0]
 12: __libc_start_main()
 13: _start()
2023-12-15T00:04:22.288 INFO:scrape:1 jobs: ['7491476']
2023-12-15T00:04:22.288 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}']
2023-12-15T00:04:22.288 INFO:scrape:
2023-12-15T00:04:22.288 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi050 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and cloud_transition'"
2023-12-15T00:04:22.288 INFO:scrape:1 jobs: ['7491458']
2023-12-15T00:04:22.288 INFO:scrape:suites: ['ignore-pg-availability', 'overrides', 'rgw/cloud-transition/{cluster', 's3tests-branch', 'supported-random-distro$/{centos_latest}', 'tasks/cloud_transition_s3tests}']
2023-12-15T00:04:22.289 INFO:scrape: