2023-12-07T21:37:50.968 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-07T21:37:50.968 INFO:teuthology.results:subset: '111/12000' 2023-12-07T21:37:50.968 INFO:teuthology.results:seed: '616' 2023-12-07T21:48:55.663 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-07T21:48:55.666 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-12-08T00:55:10.819 INFO:teuthology.results:Tests finished! gathering results... 2023-12-08T00:55:14.003 INFO:scrape:Found 43 jobs 2023-12-08T00:55:22.395 INFO:scrape:Found 5 distinct failure reasons 2023-12-08T00:55:22.396 INFO:scrape:Failure: Command failed on smithi162 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' 2023-12-08T00:55:22.396 INFO:scrape:2 jobs: ['7482433', '7482389'] 2023-12-08T00:55:22.396 INFO:scrape:suites intersection: ['overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-12-08T00:55:22.396 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-12-08T00:55:22.396 INFO:scrape: 2023-12-08T00:55:22.396 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi123 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2023-12-08T00:55:22.396 INFO:scrape:15 jobs: ['7482409', '7482415', '7482453', '7482443', '7482395', '7482437', '7482397', '7482377', '7482411', '7482439', '7482422', '7482451', '7482401', '7482455', '7482413'] 2023-12-08T00:55:22.397 INFO:scrape:suites intersection: [] 2023-12-08T00:55:22.397 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '2-kms/kmip', '2-kms/testing', '2-kms/vault_kv', '2-kms/vault_old', '2-kms/vault_transit', '3-rgw/rgw', '4-tests/{s3tests}', 'bucket-check', 'cls', 'clusters/fixed-2', 'frontend/beast', 'frontend/civetweb', 'http', 'ignore-pg-availability', 'ignore-pg-availability}', 'install', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/multifs/{clusters/fixed-2', 'rgw/thrash/{civetweb', 'rgw/verify/{centos_latest', 'rgw/website/{clusters/fixed-2', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/rgw_s3tests', 'tasks/s3tests-website', 'tasks/{0-install', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'ubuntu_latest}', 'validater/lockdep}', 'validater/valgrind}', 'versioning}', 'workload/rgw_s3tests}'] 2023-12-08T00:55:22.397 INFO:scrape: 2023-12-08T00:55:22.397 INFO:scrape:Failure: rgw multisite, test failures 2023-12-08T00:55:22.397 INFO:scrape:2 jobs: ['7482449', '7482383'] 2023-12-08T00:55:22.397 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2023-12-08T00:55:22.398 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2023-12-08T00:55:22.398 INFO:scrape: 2023-12-08T00:55:22.398 INFO:scrape:Failure: ['Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: '] 2023-12-08T00:55:22.398 INFO:scrape:2 jobs: ['7482441', '7482428'] 2023-12-08T00:55:22.398 INFO:scrape:suites intersection: [] 2023-12-08T00:55:22.399 INFO:scrape:suites union: ['clusters/fixed-2', 'frontend/civetweb', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'omap_limits', 'overrides', 'realms/three-zones', 'rgw/multisite/{clusters', 'rgw/thrash/{civetweb', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'workload/rgw_multipart_upload}'] 2023-12-08T00:55:22.399 INFO:scrape: 2023-12-08T00:55:22.399 INFO:scrape:Failure: machine smithi060.front.sepia.ceph.com is not locked 2023-12-08T00:55:22.399 INFO:scrape:1 jobs: ['7482461'] 2023-12-08T00:55:22.399 INFO:scrape:suites: ['clusters/fixed-2', 'install', 'objectstore/bluestore-bitmap', 'rgw/thrash/{civetweb', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'workload/rgw_user_quota}'] 2023-12-08T00:55:22.399 INFO:scrape: 2023-12-08T00:55:22.401 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/ failed: 20 dead: 2 running: 0 waiting: 0 queued: 0 passed: 21 Fail ================================================================= [7482377] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:23:35 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482377/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482377/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b3d67e38b0146c88c61d71ffb3f4c70 Command failed (s3 tests against rgw) on smithi124 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482383] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:23:04 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482383/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c31b7f2fb3994afebf0bb705120f08f4 rgw multisite, test failures [7482389] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:22:54 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482389/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482389/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e45f8ec67156484a9e8efe486515bcbf Command failed on smithi094 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7482395] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} ----------------------------------------------------------------- time: 00:47:55 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482395/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce44dbfaaa9047d4b49dcd9ba849c13c Command failed (s3 tests against rgw) on smithi001 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482397] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} ----------------------------------------------------------------- time: 00:20:27 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482397/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39145a721f99488aa9110f61a40f228d Command failed (s3 tests against rgw) on smithi026 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482401] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:15:45 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a7acadc3f1a4d69a5223f2f16d6e2bf Command failed (s3 tests against rgw) on smithi033 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482409] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} ----------------------------------------------------------------- time: 00:21:04 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482409/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482409/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4c7c8c334d547d88bc392468c481cf4 Command failed (s3 tests against rgw) on smithi123 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482411] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 00:39:12 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482411/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482411/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a79117fbc824001b5e476454dfb5c39 Command failed (s3 tests against rgw) on smithi118 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482413] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:14:29 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482413/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0b08c286023475a8c6460e12303d20b Command failed (s3 tests against rgw) on smithi032 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482415] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} ----------------------------------------------------------------- time: 00:18:00 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482415/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482415/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78c51130f18a4c7eaec502eb79a711ce Command failed (s3 tests against rgw) on smithi039 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482422] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:15:03 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482422/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482422/ sentry: https://sentry.ceph.com/organizations/ceph/?query=faacadb9f361422294f6499d54d89ada Command failed (s3 tests against rgw) on smithi052 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482433] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:24:01 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482433/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482433/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe98f4329493410dbc9265305cdf1997 Command failed on smithi162 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7482437] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} ----------------------------------------------------------------- time: 00:43:02 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482437/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482437/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2ade2f939a94a709a87e6c58f649fd6 Command failed (s3 tests against rgw) on smithi148 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482439] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} ----------------------------------------------------------------- time: 00:22:02 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482439/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482439/ sentry: https://sentry.ceph.com/organizations/ceph/?query=856681fef4c74d5bb588112871fd50a2 Command failed (s3 tests against rgw) on smithi003 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482443] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:17:42 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482443/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482443/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27ccf8b8b96f45c7a8479fa1320036ce Command failed (s3 tests against rgw) on smithi106 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482449] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 03:02:55 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482449/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482449/ sentry: https://sentry.ceph.com/organizations/ceph/?query=563b4ec93e414f828b994515e0cf2878 rgw multisite, test failures [7482451] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} ----------------------------------------------------------------- time: 00:20:55 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482451/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482451/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0cfc13f30ca4f1fb5404deb1e47f48e Command failed (s3 tests against rgw) on smithi073 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482453] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 00:47:27 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482453/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482453/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d03060d01b0141f3bb1b8dc5a203aeb1 Command failed (s3 tests against rgw) on smithi183 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482455] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:14:36 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482455/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482455/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cf2aa9f1d5e14212ab90de041dd9cf4d Command failed (s3 tests against rgw) on smithi115 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_s ubdomain'" [7482461] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482461/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482461/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ab71d209b964c8bb11f6434986cb29b machine smithi060.front.sepia.ceph.com is not locked Dead ================================================================= [7482428] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 00:07:35 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482428/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482428/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54b919690db64b25be42b6d640c8d40f ['Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: '] [7482441] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} ----------------------------------------------------------------- time: 00:07:32 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482441/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482441/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50c6d923b1b54d35b948377619b6fa6d ['Failed to download key at http://download.ceph.com/keys/release.asc: Request failed: '] Pass ================================================================= [7482379] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:20:47 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482379/ [7482381] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:12:11 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482381/ [7482385] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:14:22 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482385/ [7482387] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{rhel_8} tasks/{0-install first ststests}} time: 00:34:27 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482387/ [7482391] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:16:34 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482391/ [7482393] rgw/tools/{centos_latest cluster tasks} time: 03:01:34 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482393/ [7482399] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:16:22 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482399/ [7482403] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:12:18 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482403/ [7482405] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:12:41 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482405/ [7482407] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} time: 00:37:22 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482407/ [7482417] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:18:21 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482417/ [7482420] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:15:41 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482420/ [7482424] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:21:14 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482424/ [7482426] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:12:19 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482426/ [7482430] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:28:36 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482430/ [7482431] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:46:12 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482431/ [7482435] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:16:55 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482435/ [7482445] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:13:48 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482445/ [7482447] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:14:46 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482447/ [7482457] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:42 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482457/ [7482459] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:11:41 info: https://pulpito.ceph.com/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482459/