2021-10-20T05:43:18.936 INFO:root:teuthology version: 1.1.0-b35344e8 2021-10-20T05:43:18.938 INFO:teuthology.results:subset: '1/20' 2021-10-20T05:43:18.938 INFO:teuthology.results:seed: '831' 2021-10-20T05:43:35.770 INFO:root:teuthology version: 1.1.0-b35344e8 2021-10-20T05:43:35.772 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-10-20T17:44:29.475 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-10-20T17:44:32.835 INFO:scrape:Found 35 jobs 2021-10-20T17:44:35.551 INFO:scrape:Found 5 distinct failure reasons 2021-10-20T17:44:35.552 INFO:scrape:Dead: b'2021-10-20T06:32:48.655 INFO:tasks.rgw.client.0.smithi197.stdout:' 2021-10-20T17:44:35.553 INFO:scrape:1 jobs: ['6451763'] 2021-10-20T17:44:35.553 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{centos_8}}'] 2021-10-20T17:44:35.554 INFO:scrape: 2021-10-20T17:44:35.554 INFO:scrape:Failure: rgw multisite test failures 2021-10-20T17:44:35.555 INFO:scrape:1 jobs: ['6451754'] 2021-10-20T17:44:35.555 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2021-10-20T17:44:35.556 INFO:scrape: 2021-10-20T17:44:35.556 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi038 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2021-10-20T17:44:35.557 INFO:scrape:4 jobs: ['6451782', '6451757', '6451778', '6451771'] 2021-10-20T17:44:35.557 INFO:scrape:suites intersection: ['ignore-pg-availability'] 2021-10-20T17:44:35.558 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/vault_kv', '2-kms/vault_old', '2-kms/vault_transit', '3-rgw/rgw', '4-tests/{s3tests}', 'first', 'ignore-pg-availability', 'objectstore', 'overrides', 'pool-type', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/sts/{cluster', 'rgw_frontend/beast', 'ststests}}', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{centos_8.stream}}', 'supported-random-distro$/{rhel_8}}', 'tasks/{0-install'] 2021-10-20T17:44:35.558 INFO:scrape: 2021-10-20T17:44:35.559 INFO:scrape:Failure: "2021-10-20T06:07:08.527283+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log 2021-10-20T17:44:35.559 INFO:scrape:1 jobs: ['6451784'] 2021-10-20T17:44:35.560 INFO:scrape:suites: ['frontend/beast', 'objectstore/filestore-xfs', 'overrides', 'rgw/singleton/{all/radosgw-admin', 'rgw_pool_type/replicated', 'supported-random-distro$/{centos_8.stream}}'] 2021-10-20T17:44:35.560 INFO:scrape: 2021-10-20T17:44:35.561 INFO:scrape:Failure: Command failed on smithi037 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2021-10-20T17:44:35.561 INFO:scrape:1 jobs: ['6451758'] 2021-10-20T17:44:35.562 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2021-10-20T17:44:35.562 INFO:scrape: 2021-10-20T17:44:35.564 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/ failed: 7 dead: 0 running: 1 waiting: 0 queued: 0 passed: 27 Fail ================================================================= [6451754] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 04:29:31 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451754/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451754/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68f1307f7a494133b3ac7f7b0fe5d252 rgw multisite test failures [6451757] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8.stream} tasks/{0-install first ststests}} ----------------------------------------------------------------- time: 00:31:32 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451757/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451757/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce79276e83344fe79c594135868f617a Command failed (s3 tests against rgw) on smithi041 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a test_of_sts' [6451758] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:21:17 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451758/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ba3ac2483b64eab9e518896cc46184b Command failed on smithi037 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" [6451771] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8.stream}} ----------------------------------------------------------------- time: 00:16:35 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451771/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451771/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c66bcf4208394f2394fbb878731d8dfe Command failed (s3 tests against rgw) on smithi171 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6451778] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:25:18 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451778/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451778/ sentry: https://sentry.ceph.com/organizations/ceph/?query=adc2b048a1b34c099649d655b18ad583 Command failed (s3 tests against rgw) on smithi131 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6451782] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:25:58 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451782/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451782/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6cf86cb751248bca7dd0e2f29ed7e8c Command failed (s3 tests against rgw) on smithi038 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6451784] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} ----------------------------------------------------------------- time: 00:15:10 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451784/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451784/ "2021-10-20T06:07:08.527283+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log Running ================================================================= [6451763] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451763/ Pass ================================================================= [6451750] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:26 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451750/ [6451751] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:20:55 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451751/ [6451752] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:18:51 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451752/ [6451753] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:09:58 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451753/ [6451755] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:33:13 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451755/ [6451756] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:14:23 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451756/ [6451759] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:15:46 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451759/ [6451760] rgw/tools/{centos_latest cluster tasks} time: 02:45:00 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451760/ [6451761] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:19:01 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451761/ [6451762] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:14:59 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451762/ [6451764] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:10:20 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451764/ [6451765] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:12:53 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451765/ [6451766] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:59 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451766/ [6451767] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:12:25 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451767/ [6451768] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:14:29 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451768/ [6451769] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} time: 00:15:33 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451769/ [6451770] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:12:56 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451770/ [6451772] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:16:28 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451772/ [6451773] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:11:29 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451773/ [6451774] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8.stream} tasks/test_multi} time: 00:36:20 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451774/ [6451775] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:14:12 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451775/ [6451776] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:16 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451776/ [6451777] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:34 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451777/ [6451779] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:12:14 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451779/ [6451780] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:24:06 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451780/ [6451781] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:44 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451781/ [6451783] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:43 info: http://pulpito.ceph.com/ivancich-2021-10-20_05:42:42-rgw-eric-testing-1-distro-basic-smithi/6451783/