2024-02-10T02:13:58.266 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-10T02:13:58.266 INFO:teuthology.results:seed: '7513'
2024-02-10T02:14:08.840 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-10T02:14:08.841 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-10T04:05:17.351 INFO:teuthology.results:Tests finished! gathering results...
2024-02-10T04:05:17.442 INFO:scrape:Found 1 jobs
2024-02-10T04:05:18.029 INFO:scrape:Found 1 distinct failure reasons
2024-02-10T04:05:18.029 INFO:scrape:Crash: rgw multisite test failures
2024-02-10T04:05:18.029 INFO:scrape:ceph version 19.0.0-1309-g794fc37a (794fc37a67da2a8a861b302faf6d40d49eb57430) squid (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f663a91e520]
 2: /lib/librados.so.2(+0x9b7ae) [0x7f663c3f57ae]
 3: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*, int)+0x76) [0x7f663c3d96a6]
 4: (rgw_rados_operate(DoutPrefixProvider const*, librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*, optional_yield, int)+0x543) [0x55cee19a5f53]
 5: (rgw::cls::fifo::FIFO::push_entries(DoutPrefixProvider const*, std::deque<ceph::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, unsigned long, optional_yield)+0x61e) [0x55cee1ada29e]
 6: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, std::vector<ceph::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, optional_yield)+0x9ad) [0x55cee1addedd]
 7: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, ceph::buffer::v15_2_0::list const&, optional_yield)+0x13c) [0x55cee1adebec]
 8: radosgw-admin(+0x7a2bab) [0x55cee1811bab]
 9: (RGWDataChangesLog::add_entry(DoutPrefixProvider const*, RGWBucketInfo const&, rgw::bucket_log_layout_generation const&, int, optional_yield)+0x8a1) [0x55cee1816001]
 10: (add_datalog_entry(DoutPrefixProvider const*, RGWDataChangesLog*, RGWBucketInfo const&, unsigned int, optional_yield)+0x49) [0x55cee18af329]
 11: (RGWRados::Bucket::UpdateIndex::complete(DoutPrefixProvider const*, long, unsigned long, unsigned long, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*, RGWObjCategory, std::__cxx11::list<rgw_obj_index_key, std::allocator<rgw_obj_index_key> >*, optional_yield, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, bool, bool)+0x3aa) [0x55cee18dc2ea]
 12: (RGWRados::Object::Write::_do_write_meta(unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, bool, void*, req_context const&, bool)+0xa14) [0x55cee18c9964]
 13: (RGWRados::Object::Write::write_meta(unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, req_context const&, bool)+0x55b) [0x55cee18cad1b]
 14: (rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rgw_zone_set*, bool*, req_context const&, unsigned int)+0x2f7) [0x55cee18b2127]
 15: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo&, RGWBucketInfo*, std::optional<rgw_placement_rule>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional<unsigned long>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, void (*)(long, void*), void*, req_context const&, RGWFetchObjFilter*, bool, rgw_obj const&, rgw_zone_set_entry const&, rgw_zone_set*, std::optional<unsigned long>*)+0x1992) [0x55cee18d0142]
 16: (RGWAsyncFetchRemoteObj::_send_request(DoutPrefixProvider const*)+0x618) [0x55cee17ff138]
 17: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0x31) [0x55cee17ec941]
 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0x94a) [0x7f663bb80cea]
 19: /usr/lib/ceph/libceph-common.so.2(+0x261e35) [0x7f663bb7be35]
 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f663a970b43]
 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f663aa02a00]
2024-02-10T04:05:18.029 INFO:scrape:1 jobs: ['7555391']
2024-02-10T04:05:18.029 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/two-zones', 'rgw:multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2024-02-10T04:05:18.030 INFO:scrape:
2024-02-10T04:05:18.030 INFO:teuthology.results:Sending results to smanjara@redhat.com: Test Run: smanjara-2024-02-10_02:13:38-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/smanjara-2024-02-10_02:13:38-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/smanjara-2024-02-10_02:13:38-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/
failed:  1
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7555391]  rgw:multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   01:37:43
info:   https://pulpito.ceph.com/smanjara-2024-02-10_02:13:38-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7555391/
log:    http://qa-proxy.ceph.com/teuthology/smanjara-2024-02-10_02:13:38-rgw:multisite-wip-shilpa-test-wo-sync-policy-distro-default-smithi/7555391/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b1f2a1b0182b47789e921183abb20970

    rgw multisite test failures