2023-10-12T14:41:06.322 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07
2023-10-12T14:41:06.323 INFO:teuthology.results:subset: '1/2'
2023-10-12T14:41:06.323 INFO:teuthology.results:seed: '923'
2023-10-12T16:45:21.632 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07
2023-10-12T16:45:21.660 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-10-12T16:45:21.693 INFO:teuthology.results:Tests finished! gathering results...
2023-10-12T16:45:22.775 INFO:scrape:Found 12 jobs
2023-10-12T16:45:25.124 INFO:scrape:Found 3 distinct failure reasons
2023-10-12T16:45:25.125 INFO:scrape:Failure: Command failed (workunit test rgw/test_librgw_file.sh) on smithi186 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b189e13e192d61a59e1ca2678648cfd0728e56b9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'
2023-10-12T16:45:25.125 INFO:scrape:4 jobs: ['7423182', '7423188', '7423179', '7423186']
2023-10-12T16:45:25.126 INFO:scrape:suites intersection: ['cls', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests', 's3tests-branch', 's3tests-java', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'versioning}']
2023-10-12T16:45:25.126 INFO:scrape:suites union: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'validater/valgrind}', 'versioning}']
2023-10-12T16:45:25.126 INFO:scrape:
2023-10-12T16:45:25.127 INFO:scrape:Crash: Command failed (workunit test rgw/test_librgw_file.sh) on smithi055 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b189e13e192d61a59e1ca2678648cfd0728e56b9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'
2023-10-12T16:45:25.127 INFO:scrape:ceph version 18.0.0-6690-gb189e13e (b189e13e192d61a59e1ca2678648cfd0728e56b9) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f70a9054db0]
 2: (tcmalloc::CentralFreeList::ReleaseToSpans(void*)+0x60) [0x7f70a6e29170]
 3: (tcmalloc::CentralFreeList::ReleaseListToSpans(void*)+0x26) [0x7f70a6e292f6]
 4: (tcmalloc::CentralFreeList::InsertRange(void*, void*, int)+0x5c) [0x7f70a6e295ec]
 5: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x115) [0x7f70a6e297a5]
 6: (tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned int)+0x30) [0x7f70a6e29b20]
 7: /usr/lib64/ceph/rgw_sal/librgw_sal_rados.so(+0x55fe4d) [0x7f707dd5fe4d]
 8: (RGWSI_SysObj_Cache::read(DoutPrefixProvider const*, RGWSI_SysObj_Obj_GetObjState&, RGWObjVersionTracker*, rgw_raw_obj const&, ceph::buffer::v15_2_0::list*, long, long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, unsigned long*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >*, bool, rgw_cache_entry_info*, boost::optional<obj_version>, optional_yield)+0x4df) [0x7f707dd6fb6f]
 9: (RGWSI_SysObj::Obj::ROp::read(DoutPrefixProvider const*, long, long, ceph::buffer::v15_2_0::list*, optional_yield)+0xbb) [0x7f707dbd71bb]
 10: (RGWSystemMetaObj::read_info(DoutPrefixProvider const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, optional_yield, bool)+0x1f8) [0x7f707db94358]
 11: (RGWSystemMetaObj::init(DoutPrefixProvider const*, ceph::common::CephContext*, RGWSI_SysObj*, optional_yield, bool, bool)+0x7e) [0x7f707db946fe]
 12: (RGWSI_Zone::do_start(optional_yield, DoutPrefixProvider const*)+0x47a) [0x7f707dbe98da]
 13: (RGWServiceInstance::start(optional_yield, DoutPrefixProvider const*)+0x20) [0x7f707dcc31c0]
 14: (RGWSI_Notify::do_start(optional_yield, DoutPrefixProvider const*)+0x3c) [0x7f707dd6adbc]
 15: (RGWServiceInstance::start(optional_yield, DoutPrefixProvider const*)+0x20) [0x7f707dcc31c0]
 16: (RGWServices_Def::init(ceph::common::CephContext*, bool, bool, bool, optional_yield, DoutPrefixProvider const*)+0xd27) [0x7f707dccb7b7]
 17: (RGWServices::do_init(ceph::common::CephContext*, bool, bool, bool, optional_yield, DoutPrefixProvider const*)+0x2c) [0x7f707dccd3ac]
 18: (RGWRados::init_svc(bool, DoutPrefixProvider const*)+0x5b) [0x7f707dc1169b]
 19: (RGWRados::init_begin(DoutPrefixProvider const*)+0x32) [0x7f707dc170a2]
 20: new_Driver()
 21: (DriverManager::init_storage_provider(DoutPrefixProvider const*, ceph::common::CephContext*, DriverManager::Config const&, bool, bool, bool, bool, bool, bool, bool, bool, optional_yield)+0x183) [0x7f70aaaf4493]
 22: (rgw::AppMain::init_storage()+0x280) [0x7f70aa7a3d20]
 23: (rgw::RGWLib::init(std::vector<char const*, std::allocator<char const*> >&)+0x3ea) [0x7f70aa88adaa]
 24: librgw_create()
 25: (LibRGW_INIT_Test::TestBody()+0x43) [0x55fa3f38e7c3]
 26: ceph_test_librgw_file_nfsns(+0x6bfec) [0x55fa3f3cdfec]
 27: (testing::Test::Run()+0xe3) [0x55fa3f3b2323]
 28: ceph_test_librgw_file_nfsns(+0x6c1c5) [0x55fa3f3ce1c5]
 29: ceph_test_librgw_file_nfsns(+0x6c359) [0x55fa3f3ce359]
 30: (testing::internal::UnitTestImpl::RunAllTests()+0x5d6) [0x55fa3f3be9f6]
 31: (testing::UnitTest::Run()+0x1a8) [0x55fa3f3bd048]
2023-10-12T16:45:25.128 INFO:scrape:3 jobs: ['7423185', '7423177', '7423183']
2023-10-12T16:45:25.128 INFO:scrape:suites intersection: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}']
2023-10-12T16:45:25.128 INFO:scrape:suites union: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}']
2023-10-12T16:45:25.129 INFO:scrape:
2023-10-12T16:45:25.129 INFO:scrape:Crash: Command failed (workunit test rgw/test_librgw_file.sh) on smithi131 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b189e13e192d61a59e1ca2678648cfd0728e56b9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'
2023-10-12T16:45:25.130 INFO:scrape:ceph version 18.0.0-6690-gb189e13e (b189e13e192d61a59e1ca2678648cfd0728e56b9) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f9585854db0]
 2: /lib64/libtcmalloc.so.4(+0x16522) [0x7f9583616522]
 3: (tcmalloc::CentralFreeList::ReleaseListToSpans(void*)+0x26) [0x7f95836292f6]
 4: (tcmalloc::CentralFreeList::InsertRange(void*, void*, int)+0x5c) [0x7f95836295ec]
 5: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x115) [0x7f95836297a5]
 6: (tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned int)+0x30) [0x7f9583629b20]
 7: (RGWSystemMetaObj::read_id(DoutPrefixProvider const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, optional_yield)+0x2fc) [0x7f9559bb618c]
 8: (RGWZoneGroup::read_default_id(DoutPrefixProvider const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, optional_yield, bool)+0x17f) [0x7f9559ba7c5f]
 9: (RGWSystemMetaObj::init(DoutPrefixProvider const*, ceph::common::CephContext*, RGWSI_SysObj*, optional_yield, bool, bool)+0x15d) [0x7f9559ba793d]
 10: (RGWSI_Zone::do_start(optional_yield, DoutPrefixProvider const*)+0x925) [0x7f9559bff185]
 11: (RGWServiceInstance::start(optional_yield, DoutPrefixProvider const*)+0x20) [0x7f9559d29520]
 12: (RGWSI_Notify::do_start(optional_yield, DoutPrefixProvider const*)+0x3c) [0x7f9559dcf51c]
 13: (RGWServiceInstance::start(optional_yield, DoutPrefixProvider const*)+0x20) [0x7f9559d29520]
 14: (RGWServices_Def::init(ceph::common::CephContext*, bool, bool, bool, optional_yield, DoutPrefixProvider const*)+0xd27) [0x7f9559d31d47]
 15: (RGWServices::do_init(ceph::common::CephContext*, bool, bool, bool, optional_yield, DoutPrefixProvider const*)+0x2c) [0x7f9559d3393c]
 16: (RGWRados::init_svc(bool, DoutPrefixProvider const*)+0x5b) [0x7f9559c86a5b]
 17: (RGWRados::init_begin(DoutPrefixProvider const*)+0x32) [0x7f9559c86f12]
 18: new_Driver()
 19: (DriverManager::init_storage_provider(DoutPrefixProvider const*, ceph::common::CephContext*, DriverManager::Config const&, bool, bool, bool, bool, bool, bool, bool, bool, optional_yield)+0x183) [0x7f95870f4493]
 20: (rgw::AppMain::init_storage()+0x280) [0x7f9586da3d20]
 21: (rgw::RGWLib::init(std::vector<char const*, std::allocator<char const*> >&)+0x3ea) [0x7f9586e8adaa]
 22: librgw_create()
 23: (LibRGW_INIT_Test::TestBody()+0x43) [0x56090c6317c3]
 24: ceph_test_librgw_file_nfsns(+0x6bfec) [0x56090c670fec]
 25: (testing::Test::Run()+0xe3) [0x56090c655323]
 26: ceph_test_librgw_file_nfsns(+0x6c1c5) [0x56090c6711c5]
 27: ceph_test_librgw_file_nfsns(+0x6c359) [0x56090c671359]
 28: (testing::internal::UnitTestImpl::RunAllTests()+0x5d6) [0x56090c6619f6]
 29: (testing::UnitTest::Run()+0x1a8) [0x56090c660048]
 30: main()
 31: /lib64/libc.so.6(+0x3feb0) [0x7f958583feb0]
2023-10-12T16:45:25.130 INFO:scrape:2 jobs: ['7423187', '7423181']
2023-10-12T16:45:25.131 INFO:scrape:suites intersection: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}']
2023-10-12T16:45:25.131 INFO:scrape:suites union: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}']
2023-10-12T16:45:25.131 INFO:scrape: