2023-09-21T18:53:57.411 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T18:53:57.411 INFO:teuthology.results:seed: '6316'
2023-09-21T22:19:44.609 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T22:19:44.635 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-22T07:07:25.782 INFO:teuthology.results:Tests finished! gathering results...
2023-09-22T07:07:32.070 INFO:scrape:Found 70 jobs
2023-09-22T07:07:34.640 WARNING:scrape:Not-understood crash indication 2023-09-21T19:38:05.782 INFO:tasks.rgw.c1.client.0.smithi094.stderr:daemon-helper: command crashed with signal 11
2023-09-22T07:07:34.641 WARNING:scrape:Not-understood crash indication 2023-09-21T20:57:45.961 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi094.stderr: INFO: daemon-helper: command crashed with signal 11
2023-09-22T07:07:34.641 WARNING:scrape:Not-understood crash indication 
2023-09-22T07:07:34.845 WARNING:scrape:Not-understood crash indication 2023-09-21T19:38:05.782 INFO:tasks.rgw.c1.client.0.smithi094.stderr:daemon-helper: command crashed with signal 11
2023-09-22T07:07:34.845 WARNING:scrape:Not-understood crash indication 2023-09-21T20:57:45.961 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi094.stderr: INFO: daemon-helper: command crashed with signal 11
2023-09-22T07:07:34.845 WARNING:scrape:Not-understood crash indication 
2023-09-22T07:07:35.048 WARNING:scrape:Not-understood crash indication 2023-09-21T19:38:05.782 INFO:tasks.rgw.c1.client.0.smithi094.stderr:daemon-helper: command crashed with signal 11
2023-09-22T07:07:35.048 WARNING:scrape:Not-understood crash indication 2023-09-21T20:57:45.961 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi094.stderr: INFO: daemon-helper: command crashed with signal 11
2023-09-22T07:07:35.048 WARNING:scrape:Not-understood crash indication 
2023-09-22T07:07:35.259 WARNING:scrape:Not-understood crash indication 2023-09-21T19:38:05.782 INFO:tasks.rgw.c1.client.0.smithi094.stderr:daemon-helper: command crashed with signal 11
2023-09-22T07:07:35.260 WARNING:scrape:Not-understood crash indication 2023-09-21T20:57:45.961 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi094.stderr: INFO: daemon-helper: command crashed with signal 11
2023-09-22T07:07:35.260 WARNING:scrape:Not-understood crash indication 
2023-09-22T07:07:36.144 INFO:scrape:Found 9 distinct failure reasons
2023-09-22T07:07:36.144 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6a15cc18dd]
2023-09-22T07:07:36.144 INFO:scrape:ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6a1485d520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6a15cc18dd]
 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f6a15cc1a37]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x55c1c1815631]
 8: radosgw-admin(+0x857c3a) [0x55c1c181cc3a]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55c1c182329b]
 10: main()
 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6a14844d90]
 12: __libc_start_main()
 13: _start()
2023-09-22T07:07:36.145 INFO:scrape:1 jobs: ['7400266']
2023-09-22T07:07:36.145 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-09-22T07:07:36.145 INFO:scrape:
2023-09-22T07:07:36.146 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi103 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and not fails_on_dbstore'"
2023-09-22T07:07:36.146 INFO:scrape:1 jobs: ['7400233']
2023-09-22T07:07:36.146 INFO:scrape:suites: ['ignore-pg-availability', 'overrides', 'rgw/dbstore/{cluster', 's3tests-branch', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rgw_s3tests}']
2023-09-22T07:07:36.146 INFO:scrape:
2023-09-22T07:07:36.146 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f19add63157]
2023-09-22T07:07:36.146 INFO:scrape:ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f19ab254db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f19ab2a154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f19add63157]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x1632bb) [0x7f19add632bb]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x5589cd29cdd1]
 8: radosgw-admin(+0x78bada) [0x5589cd2a2ada]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5589cd2a70cb]
 10: main()
 11: /lib64/libc.so.6(+0x3feb0) [0x7f19ab23feb0]
 12: __libc_start_main()
 13: _start()
2023-09-22T07:07:36.146 INFO:scrape:1 jobs: ['7400256']
2023-09-22T07:07:36.146 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-09-22T07:07:36.147 INFO:scrape:
2023-09-22T07:07:36.147 INFO:scrape:Failure: rgw multisite test failures
2023-09-22T07:07:36.147 INFO:scrape:2 jobs: ['7400273', '7400237']
2023-09-22T07:07:36.147 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-09-22T07:07:36.147 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-09-22T07:07:36.147 INFO:scrape:
2023-09-22T07:07:36.147 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6d6b5cc8dd]
2023-09-22T07:07:36.147 INFO:scrape:ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6d6a168520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f6d6b5cc8dd]
 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f6d6b5cca37]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x5579aff0c631]
 8: radosgw-admin(+0x857c3a) [0x5579aff13c3a]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5579aff1a29b]
 10: main()
 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6d6a14fd90]
 12: __libc_start_main()
 13: _start()
2023-09-22T07:07:36.148 INFO:scrape:1 jobs: ['7400248']
2023-09-22T07:07:36.148 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-09-22T07:07:36.148 INFO:scrape:
2023-09-22T07:07:36.148 INFO:scrape:Failure: Command failed on smithi156 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev'
2023-09-22T07:07:36.148 INFO:scrape:1 jobs: ['7400253']
2023-09-22T07:07:36.148 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}']
2023-09-22T07:07:36.148 INFO:scrape:
2023-09-22T07:07:36.148 INFO:scrape:Dead: b'2023-09-22T07:00:01.617 DEBUG:teuthology.exit:Finished running handlers'
2023-09-22T07:07:36.149 INFO:scrape:1 jobs: ['7400238']
2023-09-22T07:07:36.149 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'supported-all-distro$/{centos_8}', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}']
2023-09-22T07:07:36.149 INFO:scrape:
2023-09-22T07:07:36.149 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f4ee9563157]
2023-09-22T07:07:36.149 INFO:scrape:ceph version 18.0.0-6192-g03616b43 (03616b4330080cadf20be440d92e33ca90de254a) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f4ee6a54db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f4ee6aa154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f4ee9563157]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x1632bb) [0x7f4ee95632bb]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x55eb357d5dd1]
 8: radosgw-admin(+0x78bada) [0x55eb357dbada]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55eb357e00cb]
 10: main()
 11: /lib64/libc.so.6(+0x3feb0) [0x7f4ee6a3feb0]
 12: __libc_start_main()
 13: _start()
2023-09-22T07:07:36.149 INFO:scrape:1 jobs: ['7400251']
2023-09-22T07:07:36.149 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-09-22T07:07:36.149 INFO:scrape:
2023-09-22T07:07:36.149 INFO:scrape:Failure: Command failed on smithi169 with status 127: 'cd /home/ubuntu/cephtest/keycloak-11.0.0 && cd bin && ./jboss-cli.sh --file=adapter-elytron-install-offline.cli'
2023-09-22T07:07:36.150 INFO:scrape:1 jobs: ['7400241']
2023-09-22T07:07:36.150 INFO:scrape:suites: ['1-keycloak', '2-s3tests}}', 'ignore-pg-availability', 'objectstore', 'overrides', 'pool-type', 'rgw/sts/{cluster', 'rgw_frontend/beast', 's3tests-branch', 'supported-random-distro$/{centos_latest}', 'tasks/{0-install']
2023-09-22T07:07:36.150 INFO:scrape:
2023-09-22T07:07:36.153 INFO:teuthology.results:Sending results to mkogan@redhat.com: Test Run: mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/
failed:  9
dead:    1
running: 0
waiting: 0
queued:  0
passed:  60


Fail
=================================================================
[7400233]  rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
-----------------------------------------------------------------
time:   00:19:11
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400233/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400233/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c31a261dea20415a871806b23963f49a

    Command failed (s3 tests against rgw) on smithi103 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3 and not
    fails_on_dbstore'"

[7400237]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:07:40
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400237/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400237/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5a8d3dcaf37648e39c53d448e36bf34a

    rgw multisite test failures

[7400241]  rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_latest} tasks/{0-install 1-keycloak 2-s3tests}}
-----------------------------------------------------------------
time:   00:14:27
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400241/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400241/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d6ef1442376d4c9084db277071655114

    Command failed on smithi169 with status 127: 'cd
    /home/ubuntu/cephtest/keycloak-11.0.0 && cd bin && ./jboss-cli.sh
    --file=adapter-elytron-install-offline.cli'

[7400248]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
-----------------------------------------------------------------
time:   01:48:23
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400248/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400248/

    saw valgrind issues

[7400251]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
-----------------------------------------------------------------
time:   01:01:40
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400251/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400251/

    saw valgrind issues

[7400253]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
-----------------------------------------------------------------
time:   00:11:29
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400253/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400253/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b8ddca9299054f35b96913c5d420fd6c

    Command failed on smithi156 with status 100:
    'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install
    python-dev'

[7400256]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
-----------------------------------------------------------------
time:   01:03:05
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400256/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400256/

    saw valgrind issues

[7400266]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
-----------------------------------------------------------------
time:   01:23:16
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400266/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400266/

    saw valgrind issues

[7400273]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:42:01
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400273/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400273/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cc4ea8f9fe3843cd9e29a814b21e047e

    rgw multisite test failures



Dead
=================================================================
[7400238]  rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400238/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400238/

    hit max job timeout



Pass
=================================================================
[7400230] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/cloud_transition_s3tests}
time:   00:27:39
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400230/

[7400231] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:35:32
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400231/

[7400232] rgw/d4n/{cluster ignore-pg-availability overrides supported-random-distro$/{ubuntu_latest} tasks/rgw_d4ntests}
time:   00:17:14
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400232/

[7400234] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop}
time:   00:22:38
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400234/

[7400235] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests}
time:   00:20:07
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400235/

[7400236] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:17:44
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400236/

[7400239] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest}
time:   00:16:49
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400239/

[7400240] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}
time:   00:18:58
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400240/

[7400242] rgw/tempest/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/rgw_tempest ubuntu_latest}
time:   00:24:55
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400242/

[7400243] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:17:18
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400243/

[7400244] rgw/tools/{centos_latest cluster ignore-pg-availability tasks}
time:   02:42:04
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400244/

[7400245] rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:21:18
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400245/

[7400246] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:42:10
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400246/

[7400247] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:24:44
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400247/

[7400249] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:15:46
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400249/

[7400250] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:40:45
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400250/

[7400252] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:18:20
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400252/

[7400254] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:41:27
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400254/

[7400255] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:23:39
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400255/

[7400257] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:18:20
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400257/

[7400258] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:17:47
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400258/

[7400259] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:45:10
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400259/

[7400260] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:16:17
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400260/

[7400261] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:18:57
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400261/

[7400262] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:26:01
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400262/

[7400263] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:16:48
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400263/

[7400264] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}
time:   00:18:34
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400264/

[7400265] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:35:57
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400265/

[7400267] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:15:05
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400267/

[7400268] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:43:07
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400268/

[7400269] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:14:45
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400269/

[7400270] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:16:44
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400270/

[7400271] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:25:38
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400271/

[7400272] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop}
time:   00:18:13
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400272/

[7400274] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:25:02
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400274/

[7400275] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:17:08
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400275/

[7400276] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:42:28
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400276/

[7400277] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:22:42
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400277/

[7400278] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:08:09
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400278/

[7400279] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:40:06
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400279/

[7400280] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:15:00
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400280/

[7400281] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:01:38
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400281/

[7400282] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:25:17
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400282/

[7400283] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:16:27
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400283/

[7400284] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_latest}}
time:   00:16:20
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400284/

[7400285] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:40:54
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400285/

[7400286] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:44:19
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400286/

[7400287] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:15:08
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400287/

[7400288] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:15:47
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400288/

[7400289] rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:21:12
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400289/

[7400290] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:37:54
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400290/

[7400291] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:47:31
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400291/

[7400292] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:17:38
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400292/

[7400293] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:25:43
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400293/

[7400294] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:44:43
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400294/

[7400295] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:22:59
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400295/

[7400296] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:48:35
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400296/

[7400297] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:35:20
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400297/

[7400298] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:14:22
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400298/

[7400299] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:11:13
info:   http://pulpito.ceph.com/mkogan-2023-09-21_18:53:45-rgw-rgw-wip-t62141-valg-supp_i008-distro-default-smithi/7400299/