2023-09-19T11:40:02.123 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-19T11:40:02.124 INFO:teuthology.results:subset: '1/2'
2023-09-19T11:40:02.125 INFO:teuthology.results:seed: '5056'
2023-09-19T11:41:00.739 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-19T11:41:00.740 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-19T12:09:02.714 INFO:teuthology.results:Tests finished! gathering results...
2023-09-19T12:09:03.101 INFO:scrape:Found 12 jobs
2023-09-19T12:09:03.129 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/kkeithle-2023-09-19_11:39:39-rgw:verify-wip-kkeithley-zipper-aq-distro-default-smithi/7397971/teuthology.log
2023-09-19T12:09:03.336 INFO:scrape:Found 4 distinct failure reasons
2023-09-19T12:09:03.336 INFO:scrape:Failure: No module named 'tasks.ceph'
2023-09-19T12:09:03.337 INFO:scrape:6 jobs: ['7397970', '7397973', '7397965', '7397974', '7397968', '7397975']
2023-09-19T12:09:03.337 INFO:scrape:suites intersection: ['clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests', 's3tests-branch', 's3tests-java', 'tasks/{cls', 'versioning}']
2023-09-19T12:09:03.338 INFO:scrape:suites union: ['clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}', 'versioning}']
2023-09-19T12:09:03.338 INFO:scrape:
2023-09-19T12:09:03.338 INFO:scrape:Dead: None
2023-09-19T12:09:03.339 INFO:scrape:1 jobs: ['7397971']
2023-09-19T12:09:03.339 INFO:scrape:
2023-09-19T12:09:03.339 INFO:scrape:Crash: Command failed (workunit test rgw/test_librgw_file.sh) on smithi071 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adcb54a3689e4b3aad81de4dbb06f5f1de3a1e11 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'
2023-09-19T12:09:03.340 INFO:scrape:ceph version 18.0.0-6155-gadcb54a3 (adcb54a3689e4b3aad81de4dbb06f5f1de3a1e11) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7fd5adc54db0]
 2: (tcmalloc::CentralFreeList::ReleaseToSpans(void*)+0x60) [0x7fd5aba29170]
 3: (tcmalloc::CentralFreeList::ReleaseListToSpans(void*)+0x26) [0x7fd5aba292f6]
 4: (tcmalloc::CentralFreeList::InsertRange(void*, void*, int)+0x5c) [0x7fd5aba295ec]
 5: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x115) [0x7fd5aba297a5]
 6: (tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned int)+0x30) [0x7fd5aba29b20]
 7: /usr/lib64/ceph/rgw_sal/librgw_sal_rados.so(+0x55934d) [0x7fd58655934d]
 8: (RGWSI_SysObj_Cache::read(DoutPrefixProvider const*, RGWSI_SysObj_Obj_GetObjState&, RGWObjVersionTracker*, rgw_raw_obj const&, ceph::buffer::v15_2_0::list*, long, long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, unsigned long*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >*, bool, rgw_cache_entry_info*, boost::optional<obj_version>, optional_yield)+0x4df) [0x7fd58656da8f]
 9: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0xbb) [0x7fd5863d567b]
 10: (RGWSystemMetaObj::read_info(DoutPrefixProvider const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, optional_yield, bool)+0x1f8) [0x7fd586392878]
 11: (RGWSystemMetaObj::init(DoutPrefixProvider const*, ceph::common::CephContext*, RGWSI_SysObj*, optional_yield, bool, bool)+0x7e) [0x7fd586392c1e]
 12: (RGWSI_Zone::do_start(optional_yield, DoutPrefixProvider const*)+0x47a) [0x7fd5863e7bfa]
 13: (RGWServiceInstance::start(optional_yield, DoutPrefixProvider const*)+0x20) [0x7fd5864c69a0]
 14: (RGWSI_Notify::do_start(optional_yield, DoutPrefixProvider const*)+0x3c) [0x7fd58656524c]
 15: (RGWServiceInstance::start(optional_yield, DoutPrefixProvider const*)+0x20) [0x7fd5864c69a0]
 16: (RGWServices_Def::init(ceph::common::CephContext*, bool, bool, bool, optional_yield, DoutPrefixProvider const*)+0xd27) [0x7fd5864cc157]
 17: (RGWServices::do_init(ceph::common::CephContext*, bool, bool, bool, optional_yield, DoutPrefixProvider const*)+0x2c) [0x7fd5864cdd4c]
 18: (RGWRados::init_svc(bool, DoutPrefixProvider const*)+0x5b) [0x7fd58640f24b]
 19: (RGWRados::init_begin(DoutPrefixProvider const*)+0x32) [0x7fd5864153f2]
 20: new_Driver()
 21: (DriverManager::init_storage_provider(DoutPrefixProvider const*, ceph::common::CephContext*, DriverManager::Config const&, bool, bool, bool, bool, bool, bool, bool, bool, optional_yield)+0x183) [0x7fd5af6f00c3]
 22: (rgw::AppMain::init_storage()+0x280) [0x7fd5af3a1480]
 23: (rgw::RGWLib::init(std::vector<char const*, std::allocator<char const*> >&)+0x3ea) [0x7fd5af48319a]
 24: librgw_create()
 25: (LibRGW_INIT_Test::TestBody()+0x43) [0x55eb269a17c3]
 26: ceph_test_librgw_file_nfsns(+0x6bfec) [0x55eb269e0fec]
 27: (testing::Test::Run()+0xe3) [0x55eb269c5323]
 28: ceph_test_librgw_file_nfsns(+0x6c1c5) [0x55eb269e11c5]
 29: ceph_test_librgw_file_nfsns(+0x6c359) [0x55eb269e1359]
 30: (testing::internal::UnitTestImpl::RunAllTests()+0x5d6) [0x55eb269d19f6]
 31: (testing::UnitTest::Run()+0x1a8) [0x55eb269d0048]
2023-09-19T12:09:03.340 INFO:scrape:1 jobs: ['7397976']
2023-09-19T12:09:03.341 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-09-19T12:09:03.341 INFO:scrape:
2023-09-19T12:09:03.341 INFO:scrape:Failure: HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/wip-kkeithley-zipper-aq/adcb54a3689e4b3aad81de4dbb06f5f1de3a1e11/centos/8/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')))
2023-09-19T12:09:03.342 INFO:scrape:4 jobs: ['7397967', '7397969', '7397972', '7397966']
2023-09-19T12:09:03.342 INFO:scrape:suites intersection: ['clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests', 's3tests-branch', 's3tests-java', 'tasks/{cls', 'versioning}']
2023-09-19T12:09:03.343 INFO:scrape:suites union: ['clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}', 'versioning}']
2023-09-19T12:09:03.343 INFO:scrape: