2023-03-21T17:09:53.199 INFO:root:teuthology version: 0.0.1.dev64+g923b727
2023-03-21T17:09:53.200 INFO:teuthology.results:subset: '1/2'
2023-03-21T17:09:53.201 INFO:teuthology.results:seed: '831'
2023-03-21T17:41:24.828 INFO:root:teuthology version: 0.0.1.dev64+g923b727
2023-03-21T17:41:24.857 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-03-21T18:29:28.406 INFO:teuthology.results:Tests finished! gathering results...
2023-03-21T18:29:28.811 INFO:scrape:Found 11 jobs
2023-03-21T18:29:30.588 INFO:scrape:Found 1 distinct failure reasons
2023-03-21T18:29:30.589 INFO:scrape:Crash: Command failed (s3 tests against rgw) on smithi052 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'"
2023-03-21T18:29:30.589 INFO:scrape:ceph version 18.0.0-3078-ge8457ca1 (e8457ca13a0b61517c8240951db47325730862de) reef (dev)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f16d1aaacf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f16d3905e19]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f16d3905f85]
 6: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError> const&)+0x41) [0x556d937b8331]
 7: radosgw-admin(+0xb1c3e2) [0x556d937b13e2]
 8: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x225) [0x556d937b5305]
 9: main()
 10: __libc_start_main()
 11: _start()
2023-03-21T18:29:30.589 INFO:scrape:11 jobs: ['7215490', '7215489', '7215496', '7215494', '7215491', '7215488', '7215493', '7215492', '7215486', '7215495', '7215487']
2023-03-21T18:29:30.590 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-branch', 's3tests-java', 's3tests}', 'tasks/{cls']
2023-03-21T18:29:30.590 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}']
2023-03-21T18:29:30.590 INFO:scrape:
2023-03-21T18:29:30.591 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/
failed:  11
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7215486]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:32:23
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215486/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215486/
sentry: https://sentry.ceph.com/organizations/ceph/?query=38d329fea4364314af49891c7d4f1bba

    Command failed (s3 tests against rgw) on smithi120 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215487]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   01:00:01
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215487/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215487/
sentry: https://sentry.ceph.com/organizations/ceph/?query=315277515a874f56a3b9e1786a7f0564

    Command failed (s3 tests against rgw) on smithi067 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215488]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:32:34
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215488/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215488/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c2ffbbb2aadb46bea02a264241ebf3f8

    Command failed (s3 tests against rgw) on smithi039 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215489]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   01:00:55
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215489/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215489/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1432a85b64494a82b3540939565783c8

    Command failed (s3 tests against rgw) on smithi043 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215490]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:31:36
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215490/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215490/
sentry: https://sentry.ceph.com/organizations/ceph/?query=06866fee2c704054a92fa219ad71f1af

    Command failed (s3 tests against rgw) on smithi052 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215491]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:58:38
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215491/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215491/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4a3e6c708a1d46c3b8df602b5497cf58

    Command failed (s3 tests against rgw) on smithi062 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215492]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:35:27
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215492/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215492/
sentry: https://sentry.ceph.com/organizations/ceph/?query=261a2041f0434ff8855ee077b6eef7ae

    Command failed (s3 tests against rgw) on smithi032 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215493]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:48:31
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215493/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215493/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b68353112da24f75af98ed3bb00b6553

    Command failed (s3 tests against rgw) on smithi139 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215494]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:34:48
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215494/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215494/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8e672c8d5c054550aad73fecf2ec3709

    Command failed (s3 tests against rgw) on smithi136 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215495]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   00:51:48
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215495/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215495/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3fa7b37d998a471f8da3642f7fe12856

    Command failed (s3 tests against rgw) on smithi008 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7215496]  rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:36:07
info:   http://pulpito.ceph.com/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215496/
log:    http://qa-proxy.ceph.com/teuthology/ivancich-2023-03-21_17:01:52-rgw-wip-eric-testing-1-distro-default-smithi/7215496/
sentry: https://sentry.ceph.com/organizations/ceph/?query=236547b948ca404ea8e67407facf87ae

    Command failed (s3 tests against rgw) on smithi064 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"