2022-07-16T19:13:11.578 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-16T19:13:11.580 INFO:teuthology.results:subset: '1/20' 2022-07-16T19:13:11.580 INFO:teuthology.results:seed: '831' 2022-07-16T19:47:56.205 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-16T19:47:56.234 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-17T07:43:58.035 INFO:teuthology.results:Tests finished! gathering results... 2022-07-17T07:44:01.183 INFO:scrape:Found 39 jobs 2022-07-17T07:44:03.107 INFO:scrape:Found 4 distinct failure reasons 2022-07-17T07:44:03.108 INFO:scrape:Dead: b'2022-07-17T07:32:08.757 DEBUG:teuthology.exit:Finished running handlers' 2022-07-17T07:44:03.108 INFO:scrape:1 jobs: ['6933089'] 2022-07-17T07:44:03.109 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'overrides', 'rgw/notifications/{beast', 'supported-all-distro$/{centos_8}', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}'] 2022-07-17T07:44:03.110 INFO:scrape: 2022-07-17T07:44:03.110 INFO:scrape:Failure: rgw multisite test failures 2022-07-17T07:44:03.111 INFO:scrape:2 jobs: ['6933088', '6933108'] 2022-07-17T07:44:03.111 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2022-07-17T07:44:03.112 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2022-07-17T07:44:03.112 INFO:scrape: 2022-07-17T07:44:03.113 INFO:scrape:Failure: Command failed on smithi109 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2022-07-17T07:44:03.113 INFO:scrape:1 jobs: ['6933092'] 2022-07-17T07:44:03.114 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-07-17T07:44:03.115 INFO:scrape: 2022-07-17T07:44:03.115 INFO:scrape:Failure: Cannot create secret 2022-07-17T07:44:03.115 INFO:scrape:1 jobs: ['6933082'] 2022-07-17T07:44:03.116 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{rhel_8}}'] 2022-07-17T07:44:03.116 INFO:scrape: 2022-07-17T07:44:03.119 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/ failed: 4 dead: 1 running: 0 waiting: 0 queued: 0 passed: 34 Fail ================================================================= [6933082] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:15:11 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933082/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933082/ sentry: https://sentry.ceph.com/organizations/ceph/?query=11341a8a33bf482caf65dc4b50b7e07d Cannot create secret [6933088] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 01:33:19 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933088/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933088/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b578cf493a2643ebb1b7039768d6f854 rgw multisite test failures [6933092] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:27:17 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933092/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933092/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e39b64a7f074162a00abfdda29ba9c2 Command failed on smithi109 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" [6933108] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 00:34:58 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933108/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933108/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4097a0bbec9a4472a0ce2fb214b117cf rgw multisite test failures Dead ================================================================= [6933089] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933089/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933089/ hit max job timeout Pass ================================================================= [6933080] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:14:07 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933080/ [6933081] rgw/cloud-transition/{cluster overrides supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests} time: 00:31:46 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933081/ [6933083] rgw/dbstore/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests} time: 00:13:38 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933083/ [6933084] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:16:56 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933084/ [6933085] rgw/iam/{cluster ignore-pg-availability objectstore pool-type rgw_frontend/beast supported_random_distro$/{ubuntu_latest} tasks/{0-install iamtests}} time: 00:12:57 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933085/ [6933086] rgw/lifecycle/{cluster overrides supported-random-distro$/{centos_8} tasks/rgw_s3tests} time: 00:28:45 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933086/ [6933087] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:15 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933087/ [6933090] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:14:19 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933090/ [6933091] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}} time: 00:27:58 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933091/ [6933093] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:15:42 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933093/ [6933094] rgw/tools/{centos_latest cluster tasks} time: 02:52:33 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933094/ [6933095] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:19:18 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933095/ [6933096] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:39 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933096/ [6933097] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:16:19 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933097/ [6933098] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:10:11 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933098/ [6933099] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:11:16 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933099/ [6933100] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:21:13 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933100/ [6933101] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:13:39 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933101/ [6933102] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:15:26 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933102/ [6933103] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:11:49 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933103/ [6933104] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:20 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933104/ [6933105] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:15:59 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933105/ [6933106] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:15:06 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933106/ [6933107] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:09:56 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933107/ [6933109] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:13:32 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933109/ [6933110] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:06 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933110/ [6933111] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:16:24 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933111/ [6933112] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:15:36 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933112/ [6933113] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:11:57 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933113/ [6933114] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:12:39 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933114/ [6933115] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:07 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933115/ [6933116] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:14:00 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933116/ [6933117] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:28 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933117/ [6933118] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:12:00 info: http://pulpito.ceph.com/ivancich-2022-07-16_18:44:05-rgw-wip-eric-testing-1-distro-default-smithi/6933118/