2022-02-22T00:14:47.399 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-22T00:14:47.401 INFO:teuthology.results:subset: '111/120000' 2022-02-22T00:14:47.401 INFO:teuthology.results:seed: '7446' 2022-02-22T00:34:41.395 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-22T00:34:41.434 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-22T03:18:54.361 INFO:teuthology.results:Tests finished! gathering results... 2022-02-22T03:18:58.032 INFO:scrape:Found 39 jobs 2022-02-22T03:18:59.476 INFO:scrape:Found 3 distinct failure reasons 2022-02-22T03:18:59.477 INFO:scrape:Failure: Cannot create secret 2022-02-22T03:18:59.477 INFO:scrape:1 jobs: ['6699344'] 2022-02-22T03:18:59.478 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{ubuntu_latest}}'] 2022-02-22T03:18:59.478 INFO:scrape: 2022-02-22T03:18:59.479 INFO:scrape:Failure: rgw multisite test failures 2022-02-22T03:18:59.479 INFO:scrape:1 jobs: ['6699347'] 2022-02-22T03:18:59.480 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2022-02-22T03:18:59.480 INFO:scrape: 2022-02-22T03:18:59.480 INFO:scrape:Failure: Command failed on smithi167 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2022-02-22T03:18:59.481 INFO:scrape:1 jobs: ['6699351'] 2022-02-22T03:18:59.481 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-02-22T03:18:59.482 INFO:scrape: 2022-02-22T03:18:59.484 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 36 Fail ================================================================= [6699344] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:14:17 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699344/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699344/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3baa048506646588553751d871b456b Cannot create secret [6699347] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 02:31:19 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699347/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699347/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8dca7118aff4d3b9752b72bb949d206 rgw multisite test failures [6699351] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:20:01 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699351/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699351/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4bcfce3e3bab471db94075dc7cf81939 Command failed on smithi167 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" Pass ================================================================= [6699345] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:14:51 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699345/ [6699346] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:07:58 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699346/ [6699348] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:31:17 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699348/ [6699349] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:09:13 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699349/ [6699350] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{rhel_8} tasks/{0-install first ststests}} time: 00:40:37 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699350/ [6699352] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:11:55 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699352/ [6699353] rgw/tools/{centos_latest cluster tasks} time: 02:49:36 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699353/ [6699354] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:26:08 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699354/ [6699355] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:15:55 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699355/ [6699356] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:09:59 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699356/ [6699357] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:28:22 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699357/ [6699358] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:07:14 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699358/ [6699359] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:23:38 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699359/ [6699360] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:13 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699360/ [6699361] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:49:12 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699361/ [6699362] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:13:31 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699362/ [6699363] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:12:10 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699363/ [6699364] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:09:12 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699364/ [6699365] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:10:30 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699365/ [6699366] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:14:06 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699366/ [6699367] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:13:03 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699367/ [6699368] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:07:26 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699368/ [6699369] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} time: 00:30:30 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699369/ [6699370] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:22:56 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699370/ [6699371] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:44 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699371/ [6699372] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:26:51 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699372/ [6699373] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:11:05 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699373/ [6699374] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:13:25 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699374/ [6699375] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:08:29 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699375/ [6699376] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:11:42 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699376/ [6699377] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:15:13 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699377/ [6699378] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:40:29 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699378/ [6699379] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:25:15 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699379/ [6699380] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:07:28 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699380/ [6699381] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:11:36 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699381/ [6699382] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:04 info: http://pulpito.ceph.com/yuriw-2022-02-21_18:27:13-rgw-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699382/