2024-04-14T13:45:19.248 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-14T13:45:19.248 INFO:teuthology.results:seed: '3707'
2024-04-14T13:45:19.726 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-14T13:45:19.727 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-14T14:56:25.247 INFO:teuthology.results:Tests finished! gathering results...
2024-04-14T14:56:25.349 INFO:scrape:Found 1 jobs
2024-04-14T14:56:25.974 INFO:scrape:Found 1 distinct failure reasons
2024-04-14T14:56:25.974 INFO:scrape:Crash: rgw multisite test failures
2024-04-14T14:56:25.975 INFO:scrape:ceph version 19.0.0-3087-g3c1a90b2 (3c1a90b2ba50978f65b32eed068075060ee2767b) squid (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7efe27b67520]
 2: /lib/librados.so.2(+0x9ce65) [0x7efe29659e65]
 3: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*, int, opentelemetry::v1::trace::SpanContext const*)+0x83) [0x7efe2963b903]
 4: (rgw_rados_operate(DoutPrefixProvider const*, librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*, optional_yield, int, opentelemetry::v1::trace::SpanContext const*)+0x4c2) [0x5557a11c8b12]
 5: (rgw::cls::fifo::FIFO::push_entries(DoutPrefixProvider const*, std::deque<ceph::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, unsigned long, optional_yield)+0x605) [0x5557a1349a75]
 6: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, std::vector<ceph::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, optional_yield)+0x9fe) [0x5557a134c0de]
 7: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, ceph::buffer::v15_2_0::list const&, optional_yield)+0x13c) [0x5557a134ce3c]
 8: radosgw-admin(+0x7ff3cb) [0x5557a10203cb]
 9: (RGWDataChangesLog::add_entry(DoutPrefixProvider const*, RGWBucketInfo const&, rgw::bucket_log_layout_generation const&, int, optional_yield)+0x8be) [0x5557a103afbe]
 10: (add_datalog_entry(DoutPrefixProvider const*, RGWDataChangesLog*, RGWBucketInfo const&, unsigned int, optional_yield)+0x49) [0x5557a10c8a79]
 11: (RGWRados::Bucket::UpdateIndex::complete(DoutPrefixProvider const*, long, unsigned long, unsigned long, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ACLOwner const&, RGWObjCategory, std::__cxx11::list<rgw_obj_index_key, std::allocator<rgw_obj_index_key> >*, optional_yield, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, bool, bool)+0x2e3) [0x5557a10f7603]
 12: (RGWRados::Object::Write::_do_write_meta(unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, bool, void*, req_context const&, opentelemetry::v1::trace::SpanContext&, bool)+0x9f3) [0x5557a10e41d3]
 13: (RGWRados::Object::Write::write_meta(unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, req_context const&, opentelemetry::v1::trace::SpanContext&, bool)+0x55b) [0x5557a10e55ab]
 14: (rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rgw_zone_set*, bool*, req_context const&, unsigned int)+0x3c6) [0x5557a10cba36]
 15: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo&, RGWBucketInfo*, std::optional<rgw_placement_rule>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional<unsigned long>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, void (*)(long, void*), void*, req_context const&, RGWFetchObjFilter*, bool, rgw_obj const&, rgw_zone_set_entry const&, rgw_zone_set*, std::optional<unsigned long>*)+0x1cd2) [0x5557a10eb4a2]
 16: (RGWAsyncFetchRemoteObj::_send_request(DoutPrefixProvider const*)+0x722) [0x5557a10245f2]
 17: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0x31) [0x5557a0fff131]
 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0x932) [0x7efe28dc7b22]
 19: /usr/lib/ceph/libceph-common.so.2(+0x260695) [0x7efe28dc3695]
 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7efe27bb9b43]
 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7efe27c4ba00]
2024-04-14T14:56:25.975 INFO:scrape:1 jobs: ['7655769']
2024-04-14T14:56:25.975 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/two-zones', 'rgw:multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2024-04-14T14:56:25.975 INFO:scrape:
2024-04-14T14:56:25.975 INFO:teuthology.results:Sending results to smanjara@redhat.com: Test Run: smanjara-2024-04-14_13:37:02-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/smanjara-2024-04-14_13:37:02-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/smanjara-2024-04-14_13:37:02-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/
failed:  1
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7655769]  rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   00:59:14
info:   https://pulpito.ceph.com/smanjara-2024-04-14_13:37:02-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7655769/
log:    http://qa-proxy.ceph.com/teuthology/smanjara-2024-04-14_13:37:02-rgw:multisite-wip-shilpa-multisite-tests-distro-default-smithi/7655769/
sentry: https://sentry.ceph.com/organizations/ceph/?query=caf8777815a64da5b2630daa0e32b3c7

    rgw multisite test failures