2022-06-24T21:40:52.680 INFO:root:teuthology version: 1.1.1.dev376+g082ae7ef 2022-06-24T21:40:52.683 INFO:teuthology.results:subset: '1/3' 2022-06-24T21:40:52.683 INFO:teuthology.results:seed: '0' 2022-06-24T22:03:40.400 INFO:root:teuthology version: 1.1.1.dev376+g082ae7ef 2022-06-24T22:03:40.436 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-25T00:38:53.487 INFO:teuthology.results:Tests finished! gathering results... 2022-06-25T00:38:58.680 INFO:scrape:Found 58 jobs 2022-06-25T00:38:58.941 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/prsrivas-2022-06-24_15:53:46-rgw-wip-rgw-gc-break-chain-distro-default-smithi/6897706/teuthology.log 2022-06-25T00:39:01.618 INFO:scrape:Found 5 distinct failure reasons 2022-06-25T00:39:01.619 INFO:scrape:Crash: Command crashed: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph' 2022-06-25T00:39:01.620 INFO:scrape:ceph version 17.0.0-13211-g557c20f6 (557c20f6f35b657f0dbe0a98e8ed9ef0accef3c6) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f8eef677ce0] 2: __pthread_mutex_lock() 3: sqlite3_reset() 4: (SQLDeleteStaleObjectData::Execute(DoutPrefixProvider const*, rgw::store::DBOpParams*)+0x240) [0x5573cc371b10] 5: (rgw::store::DB::ProcessOp(DoutPrefixProvider const*, std::basic_string_view >, rgw::store::DBOpParams*)+0x29b) [0x5573cc6df1cb] 6: (rgw::store::DB::delete_stale_objs(DoutPrefixProvider const*, std::__cxx11::basic_string, std::allocator > const&, unsigned int)+0xbf9) [0x5573cc6fbd39] 7: (rgw::store::DB::GC::entry()+0x2d4) [0x5573cc6fc224] 8: /lib64/libpthread.so.0(+0x81ca) [0x7f8eef66d1ca] 9: clone() 2022-06-25T00:39:01.620 INFO:scrape:1 jobs: ['6897690'] 2022-06-25T00:39:01.621 INFO:scrape:suites: ['overrides', 'rgw/dbstore/{cluster', 'supported-random-distro$/{centos_8}', 'tasks/rgw_s3tests}'] 2022-06-25T00:39:01.621 INFO:scrape: 2022-06-25T00:39:01.622 INFO:scrape:Failure: {'smithi094.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} 2022-06-25T00:39:01.622 INFO:scrape:7 jobs: ['6897715', '6897692', '6897689', '6897696', '6897697', '6897733', '6897688'] 2022-06-25T00:39:01.623 INFO:scrape:suites intersection: [] 2022-06-25T00:39:01.623 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '2-kms/testing', '3-rgw/rgw', '4-tests/{s3tests}', 'first', 'frontend/beast', 'ignore-pg-availability', 'objectstore', 'objectstore/bluestore-bitmap', 'overrides', 'pool-type', 'rgw/cloud-transition/{cluster', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/lifecycle/{cluster', 'rgw/singleton/{all/radosgw-admin', 'rgw/sts/{cluster', 'rgw_frontend/beast', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'ststests}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'tasks/cloud_transition_s3tests}', 'tasks/rgw_s3tests}', 'tasks/{0-install'] 2022-06-25T00:39:01.623 INFO:scrape: 2022-06-25T00:39:01.624 INFO:scrape:Dead: None 2022-06-25T00:39:01.624 INFO:scrape:1 jobs: ['6897706'] 2022-06-25T00:39:01.625 INFO:scrape: 2022-06-25T00:39:01.625 INFO:scrape:Failure: rgw multisite test failures 2022-06-25T00:39:01.626 INFO:scrape:2 jobs: ['6897694', '6897724'] 2022-06-25T00:39:01.626 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2022-06-25T00:39:01.627 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2022-06-25T00:39:01.627 INFO:scrape: 2022-06-25T00:39:01.628 INFO:scrape:Failure: Command failed on smithi059 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2022-06-25T00:39:01.628 INFO:scrape:1 jobs: ['6897698'] 2022-06-25T00:39:01.629 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-06-25T00:39:01.629 INFO:scrape: