2024-01-24T16:27:19.717 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-24T16:27:19.718 INFO:teuthology.results:seed: '8434'
2024-01-24T16:32:02.136 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-24T16:32:02.139 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-01-24T18:02:08.931 INFO:teuthology.results:Tests finished! gathering results...
2024-01-24T18:02:09.036 INFO:scrape:Found 1 jobs
2024-01-24T18:02:09.776 INFO:scrape:Found 1 distinct failure reasons
2024-01-24T18:02:09.776 INFO:scrape:Crash: rgw multisite test failures
2024-01-24T18:02:09.777 INFO:scrape:ceph version 19.0.0-839-ge877ac4d (e877ac4de7bff7085e0ac032b0409b4b9faface4) squid (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7fefd9654db0]
 2: /lib64/librados.so.2(+0x9edce) [0x7fefdc7f0dce]
 3: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*, int)+0x76) [0x7fefdc7d1f66]
 4: (rgw_rados_operate(DoutPrefixProvider const*, librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*, optional_yield, int)+0x553) [0x559b5ef5a5f3]
 5: (rgw::cls::fifo::FIFO::push_entries(DoutPrefixProvider const*, std::deque<ceph::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, unsigned long, optional_yield)+0x449) [0x559b5f099df9]
 6: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, std::vector<ceph::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, optional_yield)+0xd93) [0x559b5f09c6b3]
 7: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, ceph::buffer::v15_2_0::list const&, optional_yield)+0x13c) [0x559b5f09d09c]
 8: radosgw-admin(+0x68914b) [0x559b5edbc14b]
 9: (RGWDataChangesLog::add_entry(DoutPrefixProvider const*, RGWBucketInfo const&, rgw::bucket_log_layout_generation const&, int, optional_yield)+0x8b7) [0x559b5edc5b47]
 10: (add_datalog_entry(DoutPrefixProvider const*, RGWDataChangesLog*, RGWBucketInfo const&, unsigned int, optional_yield)+0x49) [0x559b5ee5dba9]
 11: (RGWRados::Bucket::UpdateIndex::complete(DoutPrefixProvider const*, long, unsigned long, unsigned long, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*, RGWObjCategory, std::__cxx11::list<rgw_obj_index_key, std::allocator<rgw_obj_index_key> >*, optional_yield, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, bool, bool)+0x3bf) [0x559b5ee8d37f]
 12: (RGWRados::Object::Write::_do_write_meta(unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, bool, void*, req_context const&, bool)+0xd79) [0x559b5ee83579]
 13: (RGWRados::Object::Write::write_meta(unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, req_context const&, bool)+0x55b) [0x559b5ee8460b]
 14: (rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rgw_zone_set*, bool*, req_context const&, unsigned int)+0x2f7) [0x559b5ee5f967]
 15: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo&, RGWBucketInfo*, std::optional<rgw_placement_rule>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional<unsigned long>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, void (*)(long, void*), void*, req_context const&, RGWFetchObjFilter*, bool, rgw_obj const&, rgw_zone_set_entry const&, rgw_zone_set*, std::optional<unsigned long>*)+0x1875) [0x559b5ee7e7a5]
 16: (RGWAsyncFetchRemoteObj::_send_request(DoutPrefixProvider const*)+0x622) [0x559b5edaff52]
 17: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0x31) [0x559b5eda6cc1]
 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0x9d3) [0x7fefdbe6c343]
 19: /usr/lib64/ceph/libceph-common.so.2(+0x267915) [0x7fefdbe67915]
 20: /lib64/libc.so.6(+0x9f802) [0x7fefd969f802]
 21: /lib64/libc.so.6(+0x3f450) [0x7fefd963f450]
2024-01-24T18:02:09.777 INFO:scrape:1 jobs: ['7531067']
2024-01-24T18:02:09.777 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/two-zones', 'rgw:multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'tasks/test_multi}']
2024-01-24T18:02:09.777 INFO:scrape:
2024-01-24T18:02:09.777 INFO:teuthology.results:Sending results to smanjara@redhat.com: Test Run: smanjara-2024-01-24_16:24:23-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/smanjara-2024-01-24_16:24:23-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/smanjara-2024-01-24_16:24:23-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/
failed:  1
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7531067]  rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{centos_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   01:20:26
info:   https://pulpito.ceph.com/smanjara-2024-01-24_16:24:23-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7531067/
log:    http://qa-proxy.ceph.com/teuthology/smanjara-2024-01-24_16:24:23-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7531067/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fb8caf5ef92f4ba59366b973881dedd9

    rgw multisite test failures