2021-04-06T04:05:14.579 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-06T04:05:14.581 INFO:teuthology.results:subset: '1/20' 2021-04-06T04:05:14.581 INFO:teuthology.results:seed: '831' 2021-04-06T04:23:06.302 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-06T04:23:06.308 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-04-06T04:58:30.707 INFO:teuthology.results:Tests finished! gathering results... 2021-04-06T04:58:32.532 INFO:scrape:Found 43 jobs 2021-04-06T04:58:33.340 INFO:scrape:Found 4 distinct failure reasons 2021-04-06T04:58:33.341 INFO:scrape:Failure: Command failed on smithi201 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds'" 2021-04-06T04:58:33.341 INFO:scrape:1 jobs: ['6023375'] 2021-04-06T04:58:33.342 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2021-04-06T04:58:33.342 INFO:scrape: 2021-04-06T04:58:33.343 INFO:scrape:Failure: too many values to unpack (expected 1) 2021-04-06T04:58:33.343 INFO:scrape:2 jobs: ['6023390', '6023383'] 2021-04-06T04:58:33.343 INFO:scrape:suites intersection: ['frontend/civetweb', 'overrides', 'rgw/singleton/{all/rgw_cephadm'] 2021-04-06T04:58:33.344 INFO:scrape:suites union: ['frontend/civetweb', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'rgw/singleton/{all/rgw_cephadm', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}'] 2021-04-06T04:58:33.344 INFO:scrape: 2021-04-06T04:58:33.345 INFO:scrape:Failure: list index out of range 2021-04-06T04:58:33.345 INFO:scrape:1 jobs: ['6023370'] 2021-04-06T04:58:33.346 INFO:scrape:suites: ['rgw/ignore-pg-availability'] 2021-04-06T04:58:33.346 INFO:scrape: 2021-04-06T04:58:33.347 INFO:scrape:Failure: rgw multisite test failures 2021-04-06T04:58:33.347 INFO:scrape:1 jobs: ['6023372'] 2021-04-06T04:58:33.347 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2021-04-06T04:58:33.348 INFO:scrape: 2021-04-06T04:58:33.350 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 38 Fail ================================================================= [6023370] rgw/ignore-pg-availability ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023370/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023370/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd60e0d8e785454792bee10322723ab5 list index out of range [6023372] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 00:33:48 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023372/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb0a62cb2ef54e09b63b0d7e391b73dc rgw multisite test failures [6023375] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:20:58 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023375/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023375/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7fc5c6eade7a4675bf90a9806f6edf90 Command failed on smithi201 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds'" [6023383] rgw/singleton/{all/rgw_cephadm frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023383/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=91afb19f0cc14e18bc823493a092fb1b too many values to unpack (expected 1) [6023390] rgw/singleton/{all/rgw_cephadm frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023390/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4098be39718e4b65b4363bef9c95f647 too many values to unpack (expected 1) Pass ================================================================= [6023367] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:10:39 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023367/ [6023368] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:19:32 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023368/ [6023369] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:14:55 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023369/ [6023371] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload} time: 00:08:45 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023371/ [6023373] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-random-distro$/{rhel_8} tasks/{0-install test_kafka}} time: 00:26:03 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023373/ [6023374] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/{0-install ststests webidentity}} time: 00:26:57 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023374/ [6023376] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:12:32 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023376/ [6023377] rgw/tools/{centos_latest cluster tasks} time: 00:20:17 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023377/ [6023378] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:16:14 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023378/ [6023379] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:12:07 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023379/ [6023380] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:13:18 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023380/ [6023381] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed} time: 00:07:57 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023381/ [6023382] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/{0-install ststests webidentity}} time: 00:26:00 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023382/ [6023384] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} time: 00:21:46 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023384/ [6023385] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:20:14 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023385/ [6023386] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:10 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023386/ [6023387] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests} time: 00:13:30 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023387/ [6023388] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/{0-install ststests webidentity}} time: 00:26:31 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023388/ [6023389] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:11:42 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023389/ [6023391] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:22 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023391/ [6023392] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:13:45 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023392/ [6023393] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota} time: 00:08:11 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023393/ [6023394] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} time: 00:31:33 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023394/ [6023395] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/{0-install ststests webidentity}} time: 00:25:59 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023395/ [6023396] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} time: 00:19:59 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023396/ [6023397] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:12:23 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023397/ [6023398] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:16:27 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023398/ [6023399] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:12:20 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023399/ [6023400] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:20 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023400/ [6023401] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota} time: 00:09:38 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023401/ [6023402] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:10:27 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023402/ [6023403] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/{0-install ststests webidentity}} time: 00:26:33 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023403/ [6023404] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} time: 00:21:18 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023404/ [6023405] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:17:33 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023405/ [6023406] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:35 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023406/ [6023407] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload} time: 00:08:18 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023407/ [6023408] rgw/sts/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/{0-install ststests webidentity}} time: 00:26:09 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023408/ [6023409] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:12:09 info: http://pulpito.ceph.com/ivancich-2021-04-06_04:01:05-rgw-wip-eric-testing-1-distro-basic-smithi/6023409/