2022-09-01T08:13:29.587 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-01T08:13:29.590 INFO:teuthology.results:subset: '1/20' 2022-09-01T08:13:29.592 INFO:teuthology.results:seed: '831' 2022-09-01T08:36:28.922 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-01T08:36:28.943 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-09-01T20:37:28.842 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-09-01T20:37:32.129 INFO:scrape:Found 38 jobs 2022-09-01T20:37:33.977 INFO:scrape:Found 4 distinct failure reasons 2022-09-01T20:37:33.978 INFO:scrape:Failure: Cannot create secret 2022-09-01T20:37:33.978 INFO:scrape:1 jobs: ['7003348'] 2022-09-01T20:37:33.979 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'supported-random-distro$/{rhel_8}}'] 2022-09-01T20:37:33.979 INFO:scrape: 2022-09-01T20:37:33.980 INFO:scrape:Failure: Command failed on smithi003 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2022-09-01T20:37:33.980 INFO:scrape:1 jobs: ['7003357'] 2022-09-01T20:37:33.981 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-09-01T20:37:33.981 INFO:scrape: 2022-09-01T20:37:33.982 INFO:scrape:Dead: b'2022-09-01T20:24:44.466 DEBUG:teuthology.exit:Finished running handlers' 2022-09-01T20:37:33.982 INFO:scrape:ceph version 17.0.0-14682-g1499844a (1499844a76895a86bbe0db67466c963a5af67f0c) quincy (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7e5301bcf0] 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x93) [0x7f7e54069523] 3: (tcmalloc::ThreadCache::Scavenge()+0x4e) [0x7f7e5406986e] 4: (RGWFetchRemoteObjCR::~RGWFetchRemoteObjCR()+0x43) [0x556b0fa25a33] 5: (ceph::common::RefCountedObject::put() const+0x1ba) [0x7f7e5244748a] 6: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x360) [0x556b0fb64630] 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x556b0fb64c31] 8: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x556b0fa14cfa] 9: main() 10: __libc_start_main() 11: _start() 2022-09-01T20:37:33.982 INFO:scrape:1 jobs: ['7003353'] 2022-09-01T20:37:33.983 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'tasks/test_multi}'] 2022-09-01T20:37:33.983 INFO:scrape: 2022-09-01T20:37:33.984 INFO:scrape:Crash: rgw multisite test failures 2022-09-01T20:37:33.984 INFO:scrape:ceph version 17.0.0-14682-g1499844a (1499844a76895a86bbe0db67466c963a5af67f0c) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f8bcc172420] 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x93) [0x7f8bcc406f83] 3: (tcmalloc::ThreadCache::Scavenge()+0x4e) [0x7f8bcc4072de] 4: (ceph::common::RefCountedObject::put() const+0x121) [0x7f8bcb6dc821] 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x339) [0x561a7f4c3d89] 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x355) [0x561a7f4c5175] 7: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x561a7f37a8da] 8: main() 9: __libc_start_main() 10: _start() 2022-09-01T20:37:33.985 INFO:scrape:1 jobs: ['7003373'] 2022-09-01T20:37:33.985 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2022-09-01T20:37:33.986 INFO:scrape: 2022-09-01T20:37:33.987 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/ failed: 3 dead: 0 running: 1 waiting: 0 queued: 0 passed: 34 Fail ================================================================= [7003348] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:19:38 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003348/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003348/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34ffe52f36dc4877aeb947813ad62c92 Cannot create secret [7003357] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:25:07 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003357/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003357/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d3ee9f551af4eb1b019e7db0df77676 Command failed on smithi003 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" [7003373] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 02:22:11 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003373/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003373/ sentry: https://sentry.ceph.com/organizations/ceph/?query=823a1c63a7b4412aaec8bbea59b398ef rgw multisite test failures Running ================================================================= [7003353] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003353/ Pass ================================================================= [7003346] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:14:16 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003346/ [7003347] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests} time: 00:31:40 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003347/ [7003349] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests} time: 00:20:06 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003349/ [7003350] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:17:21 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003350/ [7003351] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests} time: 00:24:04 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003351/ [7003352] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:52 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003352/ [7003354] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:32:29 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003354/ [7003355] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:14:07 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003355/ [7003356] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:30:50 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003356/ [7003358] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:59 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003358/ [7003359] rgw/tools/{centos_latest cluster tasks} time: 02:47:53 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003359/ [7003360] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest} time: 00:18:32 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003360/ [7003361] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:15:32 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003361/ [7003362] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} time: 00:16:28 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003362/ [7003363] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:10:07 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003363/ [7003364] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:15:59 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003364/ [7003365] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:20:28 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003365/ [7003366] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}} time: 00:13:44 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003366/ [7003367] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:15:13 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003367/ [7003368] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:11:49 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003368/ [7003369] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:32 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003369/ [7003370] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} time: 00:15:28 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003370/ [7003371] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:14:14 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003371/ [7003372] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:09:02 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003372/ [7003374] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:13:03 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003374/ [7003375] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:28 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003375/ [7003376] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:03 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003376/ [7003377] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} time: 00:15:11 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003377/ [7003378] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:11:03 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003378/ [7003379] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:11:15 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003379/ [7003380] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:44 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003380/ [7003381] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}} time: 00:13:23 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003381/ [7003382] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:09:17 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003382/ [7003383] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:10:45 info: http://pulpito.ceph.com/ivancich-2022-08-31_23:19:38-rgw-wip-eric-testing-1-distro-default-smithi/7003383/