2024-03-27T15:01:06.805 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-27T15:01:06.806 INFO:teuthology.results:subset: '111/120000' 2024-03-27T15:01:06.806 INFO:teuthology.results:seed: '5452' 2024-03-27T16:03:56.020 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-27T16:03:56.058 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-27T19:50:13.538 INFO:teuthology.results:Tests finished! gathering results... 2024-03-27T19:50:17.304 INFO:scrape:Found 41 jobs 2024-03-27T19:50:28.674 INFO:scrape:Found 7 distinct failure reasons 2024-03-27T19:50:28.674 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fdd7d9610ad] 2024-03-27T19:50:28.674 INFO:scrape:ceph version 18.2.2-1021-g7c37d80f (7c37d80f7a88f4eb561c4fe02558660a650b10ac) reef (stable) 1: /lib64/libc.so.6(+0x54db0) [0x7fdd7ae54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fdd7aea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fdd7d9610ad] 6: /usr/lib64/ceph/libceph-common.so.2(+0x161211) [0x7fdd7d961211] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x557dfb95e3a1] 8: radosgw-admin(+0x73ff52) [0x557dfb96ff52] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x249) [0x557dfb974a39] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7fdd7ae3feb0] 12: __libc_start_main() 13: _start() 2024-03-27T19:50:28.675 INFO:scrape:1 jobs: ['7626352'] 2024-03-27T19:50:28.675 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-27T19:50:28.675 INFO:scrape: 2024-03-27T19:50:28.675 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7bb81610ad] 2024-03-27T19:50:28.675 INFO:scrape:ceph version 18.2.2-1021-g7c37d80f (7c37d80f7a88f4eb561c4fe02558660a650b10ac) reef (stable) 1: /lib64/libc.so.6(+0x54db0) [0x7f7bb5654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f7bb56a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7bb81610ad] 6: /usr/lib64/ceph/libceph-common.so.2(+0x161211) [0x7f7bb8161211] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5555583af3a1] 8: radosgw-admin(+0x73ff52) [0x5555583c0f52] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x249) [0x5555583c5a39] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7f7bb563feb0] 12: __libc_start_main() 13: _start() 2024-03-27T19:50:28.675 INFO:scrape:1 jobs: ['7626376'] 2024-03-27T19:50:28.675 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2024-03-27T19:50:28.675 INFO:scrape: 2024-03-27T19:50:28.675 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi043 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" 2024-03-27T19:50:28.676 INFO:scrape:9 jobs: ['7626347', '7626380', '7626343', '7626356', '7626381', '7626371', '7626353', '7626378', '7626349'] 2024-03-27T19:50:28.676 INFO:scrape:suites intersection: ['ignore-pg-availability', 's3tests-branch'] 2024-03-27T19:50:28.676 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '2-kms/testing', '2-kms/vault_kv', '2-kms/vault_old', '2-kms/vault_transit', '3-rgw/rgw', '4-tests/{s3tests}', 'frontend/beast', 'http', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/multifs/{clusters/fixed-2', 'rgw/thrash/{clusters/fixed-2', 'rgw/website/{clusters/fixed-2', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 's3tests-branch', 'tasks/rgw_s3tests', 'tasks/s3tests-website', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'ubuntu_latest}', 'workload/rgw_s3tests}'] 2024-03-27T19:50:28.676 INFO:scrape: 2024-03-27T19:50:28.676 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f92f78ca81d] 2024-03-27T19:50:28.676 INFO:scrape:ceph version 18.2.2-1021-g7c37d80f (7c37d80f7a88f4eb561c4fe02558660a650b10ac) reef (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f92f6481520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f92f78ca81d] 6: /usr/lib/ceph/libceph-common.so.2(+0x162977) [0x7f92f78ca977] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55d273487ac1] 8: radosgw-admin(+0x80464a) [0x55d27349164a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x239) [0x55d273495ac9] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f92f6468d90] 12: __libc_start_main() 13: _start() 2024-03-27T19:50:28.676 INFO:scrape:1 jobs: ['7626370'] 2024-03-27T19:50:28.676 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-27T19:50:28.676 INFO:scrape: 2024-03-27T19:50:28.676 INFO:scrape:Failure: Command failed on smithi029 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev' 2024-03-27T19:50:28.676 INFO:scrape:1 jobs: ['7626373'] 2024-03-27T19:50:28.677 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}'] 2024-03-27T19:50:28.677 INFO:scrape: 2024-03-27T19:50:28.677 INFO:scrape:Assertion: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcbdd220de5] 2024-03-27T19:50:28.677 INFO:scrape:ceph version 18.2.2-1021-g7c37d80f (7c37d80f7a88f4eb561c4fe02558660a650b10ac) reef (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fcbdb3c5ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcbdd220de5] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2aaf51) [0x7fcbdd220f51] 6: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5581f2cbc3b1] 7: radosgw-admin(+0xb383d2) [0x5581f2cb53d2] 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x225) [0x5581f2cb9305] 9: main() 10: __libc_start_main() 11: _start() 2024-03-27T19:50:28.677 INFO:scrape:1 jobs: ['7626345'] 2024-03-27T19:50:28.677 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{rhel_8}', 'tasks/{bucket-check', 'validater/lockdep}', 'versioning}'] 2024-03-27T19:50:28.677 INFO:scrape: 2024-03-27T19:50:28.677 INFO:scrape:Failure: rgw multisite test failures 2024-03-27T19:50:28.677 INFO:scrape:2 jobs: ['7626342', '7626361'] 2024-03-27T19:50:28.677 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2024-03-27T19:50:28.677 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2024-03-27T19:50:28.678 INFO:scrape: 2024-03-27T19:50:28.679 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/ failed: 16 dead: 0 running: 0 waiting: 0 queued: 0 passed: 25 Fail ================================================================= [7626342] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 01:53:35 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626342/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=414999e440fd42c19da271783c31546c rgw multisite test failures [7626343] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} ----------------------------------------------------------------- time: 00:22:45 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626343/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b442b3d9967c436eae108956f59421ed Command failed (s3 tests against rgw) on smithi069 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626345] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} ----------------------------------------------------------------- time: 00:37:04 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626345/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626345/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f5587b144b245b0a454ba0ad8251ef5 Command failed (s3 tests against rgw) on smithi115 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626347] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} ----------------------------------------------------------------- time: 00:19:12 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626347/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626347/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa8eb1035fa145caa8b739d7bf2534e1 Command failed (s3 tests against rgw) on smithi043 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626349] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest} ----------------------------------------------------------------- time: 00:20:19 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626349/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca7dac90c3064e99ae40dd18ab4627aa Command failed (s3 tests against rgw) on smithi023 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626352] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 00:46:17 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626352/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626352/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93ed5d2de42242998e4155313a1351b4 Command failed (s3 tests against rgw) on smithi175 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626353] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} ----------------------------------------------------------------- time: 00:18:59 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626353/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626353/ sentry: https://sentry.ceph.com/organizations/ceph/?query=562a52df35054aaf9e66a91723cacf82 Command failed (s3 tests against rgw) on smithi104 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain'" [7626356] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} ----------------------------------------------------------------- time: 00:28:06 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626356/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f5fca145b854fe3a9afb5dd809486e5 Command failed (s3 tests against rgw) on smithi084 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626361] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_20.04} tasks/test_multi} ----------------------------------------------------------------- time: 04:03:39 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626361/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626361/ sentry: https://sentry.ceph.com/organizations/ceph/?query=003253ec77e44896a796324630969e45 rgw multisite test failures [7626370] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} ----------------------------------------------------------------- time: 00:43:18 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626370/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626370/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d665f4fe86ea4bd89956e36e7219edb9 Command failed (s3 tests against rgw) on smithi007 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626371] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest} ----------------------------------------------------------------- time: 00:22:22 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626371/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626371/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee70bfb348bf423aa70d582bd29b3601 Command failed (s3 tests against rgw) on smithi163 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626373] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} ----------------------------------------------------------------- time: 00:10:22 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626373/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626373/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9691c1a04b948beaf19d4b382fe65e0 Command failed on smithi029 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev' [7626376] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 00:47:23 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626376/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626376/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7337076e3a64b71b0eb23a18a915973 Command failed (s3 tests against rgw) on smithi083 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626378] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} ----------------------------------------------------------------- time: 00:18:31 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626378/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626378/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac3efd797d5e43abb5a1b4d7fd63e521 Command failed (s3 tests against rgw) on smithi110 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626380] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest} ----------------------------------------------------------------- time: 00:19:50 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626380/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4469c06d11274b8c84a488b732f4bfa2 Command failed (s3 tests against rgw) on smithi107 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" [7626381] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} ----------------------------------------------------------------- time: 00:18:58 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626381/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f48b21445ac34883b568ed9d83954038 Command failed (s3 tests against rgw) on smithi134 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" Pass ================================================================= [7626341] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop} time: 00:20:15 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626341/ [7626344] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:13:42 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626344/ [7626346] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:11:28 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626346/ [7626348] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_20.04}} time: 00:11:53 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626348/ [7626350] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:15:34 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626350/ [7626351] rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:19:35 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626351/ [7626354] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:11:58 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626354/ [7626355] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/cloud_transition_s3tests} time: 00:32:32 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626355/ [7626357] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests} time: 00:21:45 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626357/ [7626358] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop} time: 00:20:42 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626358/ [7626359] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests} time: 00:24:26 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626359/ [7626360] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:14:44 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626360/ [7626362] rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:36:16 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626362/ [7626363] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest} time: 00:16:10 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626363/ [7626364] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:18:12 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626364/ [7626365] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install first ststests}} time: 00:30:14 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626365/ [7626366] rgw/tempest/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/rgw_tempest ubuntu_latest} time: 00:24:49 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626366/ [7626367] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:16:06 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626367/ [7626368] rgw/tools/{centos_latest cluster ignore-pg-availability tasks} time: 02:44:34 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626368/ [7626369] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:13:45 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626369/ [7626372] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:12:38 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626372/ [7626374] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:15:58 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626374/ [7626375] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:17:05 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626375/ [7626377] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:11:13 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626377/ [7626379] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:13:01 info: https://pulpito.ceph.com/yuriw-2024-03-27_14:57:57-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7626379/