2023-03-21T14:20:57.330 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3 2023-03-21T14:20:57.331 INFO:teuthology.results:seed: '9297' 2023-03-21T14:34:42.016 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3 2023-03-21T14:34:42.019 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-21T16:22:50.346 INFO:teuthology.results:Tests finished! gathering results... 2023-03-21T16:22:50.699 INFO:scrape:Found 10 jobs 2023-03-21T16:22:51.781 INFO:scrape:Found 2 distinct failure reasons 2023-03-21T16:22:51.782 INFO:scrape:Crash: "1679410664.046039 mon.a (mon.0) 363 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-03-21T16:22:51.782 INFO:scrape:ceph version 18.0.0-3069-g0771ac71 (0771ac71396b699035ebcf7d50e6b8c7a045f477) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f04a98d8520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f04aacff73b] 6: /usr/lib/ceph/libceph-common.so.2(+0x161895) [0x7f04aacff895] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55fc43df0a01] 8: radosgw-admin(+0x7f859a) [0x55fc43df759a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x239) [0x55fc43dfc4b9] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f04a98bfd90] 12: __libc_start_main() 13: _start() 2023-03-21T16:22:51.782 INFO:scrape:9 jobs: ['7215320', '7215321', '7215314', '7215317', '7215312', '7215316', '7215315', '7215313', '7215319'] 2023-03-21T16:22:51.783 INFO:scrape:suites intersection: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests-branch', 's3tests-java', 's3tests}', 'tasks/{cls', 'ubuntu_latest'] 2023-03-21T16:22:51.783 INFO:scrape:suites union: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'ubuntu_latest', 'validater/lockdep}', 'validater/valgrind}'] 2023-03-21T16:22:51.783 INFO:scrape: 2023-03-21T16:22:51.784 INFO:scrape:Crash: "1679410572.1392555 mon.a (mon.0) 347 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-03-21T16:22:51.784 INFO:scrape:ceph version 18.0.0-3069-g0771ac71 (0771ac71396b699035ebcf7d50e6b8c7a045f477) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f44c4b30520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f44c5f5773b] 6: /usr/lib/ceph/libceph-common.so.2(+0x161895) [0x7f44c5f57895] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5579b5df8a01] 8: radosgw-admin(+0x7f859a) [0x5579b5dff59a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x239) [0x5579b5e044b9] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f44c4b17d90] 12: __libc_start_main() 13: _start() 2023-03-21T16:22:51.784 INFO:scrape:1 jobs: ['7215318'] 2023-03-21T16:22:51.784 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'ubuntu_latest', 'validater/lockdep}'] 2023-03-21T16:22:51.785 INFO:scrape: 2023-03-21T16:22:51.785 INFO:teuthology.results:Sending results to mkogan@redhat.com: Test Run: mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/ failed: 10 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7215312] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/valgrind} ----------------------------------------------------------------- time: 01:42:07 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215312/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215312/ "1679409940.0368268 mon.a (mon.0) 362 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7215313] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/valgrind} ----------------------------------------------------------------- time: 01:39:51 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215313/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215313/ "1679410186.1044288 mon.a (mon.0) 351 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7215314] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/valgrind} ----------------------------------------------------------------- time: 01:43:48 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215314/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215314/ "1679410394.0360653 mon.a (mon.0) 331 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7215315] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/valgrind} ----------------------------------------------------------------- time: 01:09:18 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215315/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215315/ "1679410559.5500991 mon.a (mon.0) 365 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7215316] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/valgrind} ----------------------------------------------------------------- time: 01:15:25 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215316/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215316/ "1679410642.87486 mon.a (mon.0) 350 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7215317] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/valgrind} ----------------------------------------------------------------- time: 01:14:59 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215317/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215317/ "1679410640.5692308 mon.a (mon.0) 318 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7215318] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/lockdep} ----------------------------------------------------------------- time: 00:38:44 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215318/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215318/ "1679410572.1392555 mon.a (mon.0) 347 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7215319] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/lockdep} ----------------------------------------------------------------- time: 00:41:17 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215319/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215319/ "1679410636.3751314 mon.a (mon.0) 340 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7215320] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/lockdep} ----------------------------------------------------------------- time: 00:39:48 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215320/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215320/ "1679410664.046039 mon.a (mon.0) 363 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7215321] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_latest validater/lockdep} ----------------------------------------------------------------- time: 00:43:43 info: http://pulpito.ceph.com/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215321/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-03-21_14:16:48-rgw:verify-wip-ubu-SigDoesNotMatch-i01-distro-default-smithi/7215321/ "1679410694.9507353 mon.a (mon.0) 433 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log