2022-09-06T20:26:51.736 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-06T20:26:51.737 INFO:teuthology.results:subset: '1/2'
2022-09-06T20:26:51.738 INFO:teuthology.results:seed: '545'
2022-09-06T21:04:18.337 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-06T21:04:18.361 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-09-07T09:05:17.650 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2022-09-07T09:05:26.587 INFO:scrape:Found 71 jobs
2022-09-07T09:05:29.072 INFO:scrape:Found 5 distinct failure reasons
2022-09-07T09:05:29.073 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1e5388db785fcb9fd729e7ca3ad187a81d7a4f12
2022-09-07T09:05:29.074 INFO:scrape:1 jobs: ['7014570']
2022-09-07T09:05:29.074 INFO:scrape:suites: ['hadoop/v32', 'overrides', 'rgw/hadoop-s3a/{clusters/fixed-2', 's3a-hadoop}']
2022-09-07T09:05:29.075 INFO:scrape:
2022-09-07T09:05:29.075 INFO:scrape:Dead: b'2022-09-07T08:36:02.070 DEBUG:teuthology.exit:Finished running handlers'
2022-09-07T09:05:29.076 INFO:scrape:ceph version 17.0.0-14828-g1e5388db (1e5388db785fcb9fd729e7ca3ad187a81d7a4f12) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f1e3ec1f420]
 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x11f) [0x7f1e3e79575f]
 3: (std::_Rb_tree<rgw_obj_key, std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry>, std::_Select1st<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> >, std::less<rgw_obj_key>, std::allocator<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> >, std::_Rb_tree_const_iterator<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> >)+0x49) [0x557da0e26d99]
 4: (RGWBucketSyncSingleEntryCR<rgw_obj_key, rgw_obj_key>::operate(DoutPrefixProvider const*)+0x8fe) [0x557da0e4d1ae]
 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x17c) [0x557da0f4143c]
 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x355) [0x557da0f429e5]
 7: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x557da0df82ba]
 8: main()
 9: __libc_start_main()
 10: _start()
2022-09-07T09:05:29.076 INFO:scrape:1 jobs: ['7014535']
2022-09-07T09:05:29.077 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2022-09-07T09:05:29.077 INFO:scrape:
2022-09-07T09:05:29.077 INFO:scrape:Failure: Cannot create secret
2022-09-07T09:05:29.078 INFO:scrape:1 jobs: ['7014530']
2022-09-07T09:05:29.078 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'supported-random-distro$/{ubuntu_latest}}']
2022-09-07T09:05:29.079 INFO:scrape:
2022-09-07T09:05:29.079 INFO:scrape:Crash: rgw multisite test failures
2022-09-07T09:05:29.080 INFO:scrape:ceph version 17.0.0-14828-g1e5388db (1e5388db785fcb9fd729e7ca3ad187a81d7a4f12) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f0092c78420]
 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x93) [0x7f0092f0cf83]
 3: (tcmalloc::ThreadCache::Scavenge()+0x4e) [0x7f0092f0d2de]
 4: (ceph::common::RefCountedObject::put() const+0x121) [0x7f00921e17a1]
 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x339) [0x559934c015f9]
 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x355) [0x559934c029e5]
 7: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x559934ab82ba]
 8: main()
 9: __libc_start_main()
 10: _start()
2022-09-07T09:05:29.080 INFO:scrape:1 jobs: ['7014571']
2022-09-07T09:05:29.081 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2022-09-07T09:05:29.081 INFO:scrape:
2022-09-07T09:05:29.081 INFO:scrape:Failure: Command failed on smithi071 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'"
2022-09-07T09:05:29.082 INFO:scrape:1 jobs: ['7014539']
2022-09-07T09:05:29.082 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2022-09-07T09:05:29.083 INFO:scrape: