2023-09-18T18:30:04.878 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-18T18:30:04.879 INFO:teuthology.results:seed: '4720' 2023-09-18T18:31:03.907 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-18T18:31:03.911 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-18T20:30:12.005 INFO:teuthology.results:Tests finished! gathering results... 2023-09-18T20:30:12.365 INFO:scrape:Found 12 jobs 2023-09-18T20:30:12.365 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397720/teuthology.log 2023-09-18T20:30:12.633 INFO:scrape:Found 3 distinct failure reasons 2023-09-18T20:30:12.633 INFO:scrape:Dead: None 2023-09-18T20:30:12.634 INFO:scrape:1 jobs: ['7397720'] 2023-09-18T20:30:12.634 INFO:scrape: 2023-09-18T20:30:12.635 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6f0e0298dd] 2023-09-18T20:30:12.635 INFO:scrape:ceph version 18.0.0-5987-g8d652567 (8d652567c72b8e8d2e3146b3aed274fc27ae7589) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6f0cbdf520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6f0e0298dd] 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f6f0e029a37] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55ae4b537351] 8: radosgw-admin(+0x848881) [0x55ae4b550881] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55ae4b557c4b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6f0cbc6d90] 12: __libc_start_main() 13: _start() 2023-09-18T20:30:12.635 INFO:scrape:1 jobs: ['7397716'] 2023-09-18T20:30:12.636 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}'] 2023-09-18T20:30:12.636 INFO:scrape: 2023-09-18T20:30:12.636 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff7892368dd] 2023-09-18T20:30:12.637 INFO:scrape:ceph version 18.0.0-5987-g8d652567 (8d652567c72b8e8d2e3146b3aed274fc27ae7589) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff787dec520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff7892368dd] 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7ff789236a37] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x560d1c5d4351] 8: radosgw-admin(+0x848881) [0x560d1c5ed881] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x560d1c5f4c4b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ff787dd3d90] 12: __libc_start_main() 13: _start() 2023-09-18T20:30:12.637 INFO:scrape:1 jobs: ['7397722'] 2023-09-18T20:30:12.637 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}'] 2023-09-18T20:30:12.638 INFO:scrape: 2023-09-18T20:30:12.638 INFO:teuthology.results:Sending results to gsalomon@redhat.com: Test Run: gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/ failed: 2 dead: 1 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [7397716] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 01:46:04 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397716/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397716/ saw valgrind issues [7397722] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 01:23:56 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397722/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397722/ saw valgrind issues Dead ================================================================= [7397720] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397720/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397720/ Error reimaging machines: Timeout opening channel. Pass ================================================================= [7397715] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:37:14 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397715/ [7397717] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:34:12 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397717/ [7397718] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:46:19 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397718/ [7397719] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:36:50 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397719/ [7397721] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:38:12 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397721/ [7397723] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:36:16 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397723/ [7397724] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 00:56:18 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397724/ [7397725] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:39:20 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397725/ [7397726] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:05:48 info: http://pulpito.ceph.com/gsalomon-2023-09-18_18:29:49-rgw:verify-QE_defect_fixes-distro-default-smithi/7397726/