2022-09-13T13:15:12.074 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-13T13:15:12.075 INFO:teuthology.results:seed: '9036'
2022-09-13T13:31:01.397 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-13T13:31:01.428 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-09-14T01:31:59.986 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2022-09-14T01:32:00.180 INFO:scrape:Found 2 jobs
2022-09-14T01:32:01.233 INFO:scrape:Found 2 distinct failure reasons
2022-09-14T01:32:01.234 INFO:scrape:Crash: rgw multisite test failures
2022-09-14T01:32:01.234 INFO:scrape:ceph version 17.0.0-14925-g2dd12b3f (2dd12b3fa5440a4bc98a44fa2009e41893368c64) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f904e1fc420]
 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0xc2) [0x7f904dd728a2]
 3: (std::_Rb_tree<rgw_obj_key, std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry>, std::_Select1st<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> >, std::less<rgw_obj_key>, std::allocator<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> >, std::_Rb_tree_const_iterator<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> >)+0x49) [0x557a553ca849]
 4: (RGWBucketSyncSingleEntryCR<rgw_obj_key, rgw_obj_key>::operate(DoutPrefixProvider const*)+0x8fe) [0x557a553f0c5e]
 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x17c) [0x557a554e4eec]
 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x355) [0x557a554e6495]
 7: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x557a5539bd6a]
 8: main()
 9: __libc_start_main()
 10: _start()
2022-09-14T01:32:01.235 INFO:scrape:1 jobs: ['7030574']
2022-09-14T01:32:01.236 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'rgw:multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2022-09-14T01:32:01.236 INFO:scrape:
2022-09-14T01:32:01.237 INFO:scrape:Dead: b'2022-09-14T01:31:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2022-09-14T01:31:04.590+0000 7fc978026dc0  0 monclient(hunting): authenticate timed out after 300'
2022-09-14T01:32:01.237 INFO:scrape:ceph version 17.0.0-14925-g2dd12b3f (2dd12b3fa5440a4bc98a44fa2009e41893368c64) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f087f9b4420]
 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x93) [0x7f087fc48f83]
 3: (tcmalloc::ThreadCache::Scavenge()+0x4e) [0x7f087fc492de]
 4: (RGWCoroutinesStack::unwind(int)+0x5d) [0x564858493a8d]
 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x32e) [0x56485849409e]
 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x355) [0x564858495495]
 7: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x56485834ad6a]
 8: main()
 9: __libc_start_main()
 10: _start()
2022-09-14T01:32:01.238 INFO:scrape:1 jobs: ['7030573']
2022-09-14T01:32:01.238 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw:multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2022-09-14T01:32:01.238 INFO:scrape: