2024-02-18T16:44:49.033 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-18T16:44:49.033 INFO:teuthology.results:seed: '0'
2024-02-18T17:34:26.718 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-18T17:34:26.721 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-18T20:52:42.495 INFO:teuthology.results:Tests finished! gathering results...
2024-02-18T20:52:48.357 INFO:scrape:Found 71 jobs
2024-02-18T20:52:50.111 WARNING:scrape:Not-understood crash indication 2024-02-18T17:20:14.219 INFO:tasks.rgw.client.0.smithi029.stderr:daemon-helper: command crashed with signal 6
2024-02-18T20:52:50.111 WARNING:scrape:Not-understood crash indication 
2024-02-18T20:52:50.162 WARNING:scrape:Not-understood crash indication 2024-02-18T17:20:14.219 INFO:tasks.rgw.client.0.smithi029.stderr:daemon-helper: command crashed with signal 6
2024-02-18T20:52:50.162 WARNING:scrape:Not-understood crash indication 
2024-02-18T20:52:50.214 WARNING:scrape:Not-understood crash indication 2024-02-18T17:20:14.219 INFO:tasks.rgw.client.0.smithi029.stderr:daemon-helper: command crashed with signal 6
2024-02-18T20:52:50.214 WARNING:scrape:Not-understood crash indication 
2024-02-18T20:52:50.266 WARNING:scrape:Not-understood crash indication 2024-02-18T17:20:14.219 INFO:tasks.rgw.client.0.smithi029.stderr:daemon-helper: command crashed with signal 6
2024-02-18T20:52:50.266 WARNING:scrape:Not-understood crash indication 
2024-02-18T20:52:50.315 WARNING:scrape:Not-understood crash indication 2024-02-18T17:20:14.219 INFO:tasks.rgw.client.0.smithi029.stderr:daemon-helper: command crashed with signal 6
2024-02-18T20:52:50.316 WARNING:scrape:Not-understood crash indication 
2024-02-18T20:52:50.316 INFO:scrape:Found 3 distinct failure reasons
2024-02-18T20:52:50.316 INFO:scrape:Crash: rgw multisite test failures
2024-02-18T20:52:50.316 INFO:scrape:ceph version 19.0.0-1530-gd3ebf038 (d3ebf03868e5eb2fb2de3ba8da05f9affa5a2a17) squid (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ffa4f93c520]
 2: /lib/librados.so.2(+0x9b7ae) [0x7ffa514177ae]
 3: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*, int)+0x76) [0x7ffa513fb6a6]
 4: (rgw_rados_operate(DoutPrefixProvider const*, librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*, optional_yield, int)+0x543) [0x563c6131ee53]
 5: (rgw::cls::fifo::FIFO::push_entries(DoutPrefixProvider const*, std::deque<ceph::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, unsigned long, optional_yield)+0x61e) [0x563c61456b1e]
 6: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, std::vector<ceph::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> > const&, optional_yield)+0x9ad) [0x563c6145a74d]
 7: (rgw::cls::fifo::FIFO::push(DoutPrefixProvider const*, ceph::buffer::v15_2_0::list const&, optional_yield)+0x13c) [0x563c6145b45c]
 8: radosgw-admin(+0x7a6c4b) [0x563c6118ac4b]
 9: (RGWDataChangesLog::add_entry(DoutPrefixProvider const*, RGWBucketInfo const&, rgw::bucket_log_layout_generation const&, int, optional_yield)+0x8a1) [0x563c6118f0a1]
 10: (add_datalog_entry(DoutPrefixProvider const*, RGWDataChangesLog*, RGWBucketInfo const&, unsigned int, optional_yield)+0x49) [0x563c61228209]
 11: (RGWRados::Bucket::UpdateIndex::complete(DoutPrefixProvider const*, long, unsigned long, unsigned long, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*, RGWObjCategory, std::__cxx11::list<rgw_obj_index_key, std::allocator<rgw_obj_index_key> >*, optional_yield, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, bool, bool)+0x3aa) [0x563c6125528a]
 12: (RGWRados::Object::Write::_do_write_meta(unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, bool, void*, req_context const&, bool)+0xa14) [0x563c61242884]
 13: (RGWRados::Object::Write::write_meta(unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, req_context const&, bool)+0x55b) [0x563c61243c3b]
 14: (rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rgw_zone_set*, bool*, req_context const&, unsigned int)+0x2f7) [0x563c6122b017]
 15: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo&, RGWBucketInfo*, std::optional<rgw_placement_rule>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional<unsigned long>, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, void (*)(long, void*), void*, req_context const&, RGWFetchObjFilter*, bool, rgw_obj const&, rgw_zone_set_entry const&, rgw_zone_set*, std::optional<unsigned long>*)+0x1992) [0x563c61249062]
 16: (RGWAsyncFetchRemoteObj::_send_request(DoutPrefixProvider const*)+0x618) [0x563c611781d8]
 17: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0x31) [0x563c611659e1]
 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0x94a) [0x7ffa50b9ee6a]
 19: /usr/lib/ceph/libceph-common.so.2(+0x261fb5) [0x7ffa50b99fb5]
 20: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ffa4f98eb43]
 21: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ffa4fa20a00]
2024-02-18T20:52:50.316 INFO:scrape:1 jobs: ['7562822']
2024-02-18T20:52:50.317 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2024-02-18T20:52:50.317 INFO:scrape:
2024-02-18T20:52:50.317 INFO:scrape:Failure: rgw multisite test failures
2024-02-18T20:52:50.317 INFO:scrape:1 jobs: ['7562857']
2024-02-18T20:52:50.317 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'tasks/test_multi}']
2024-02-18T20:52:50.317 INFO:scrape:
2024-02-18T20:52:50.318 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi029 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test'
2024-02-18T20:52:50.318 INFO:scrape:1 jobs: ['7562823']
2024-02-18T20:52:50.318 INFO:scrape:suites: ['bluestore-bitmap', 'centos_latest', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}']
2024-02-18T20:52:50.318 INFO:scrape: