2022-10-12T16:20:05.630 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-12T16:20:05.632 INFO:teuthology.results:subset: '111/120000' 2022-10-12T16:20:05.632 INFO:teuthology.results:seed: '9216' 2022-10-12T17:09:06.525 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-12T17:09:06.553 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-12T19:19:15.912 INFO:teuthology.results:Tests finished! gathering results... 2022-10-12T19:19:18.936 INFO:scrape:Found 40 jobs 2022-10-12T19:19:20.346 INFO:scrape:Found 4 distinct failure reasons 2022-10-12T19:19:20.347 INFO:scrape:Failure: Command failed on smithi084 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2022-10-12T19:19:20.347 INFO:scrape:1 jobs: ['7063460'] 2022-10-12T19:19:20.348 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-10-12T19:19:20.348 INFO:scrape: 2022-10-12T19:19:20.349 INFO:scrape:Failure: rgw multisite test failures 2022-10-12T19:19:20.349 INFO:scrape:1 jobs: ['7063456'] 2022-10-12T19:19:20.350 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2022-10-12T19:19:20.350 INFO:scrape: 2022-10-12T19:19:20.351 INFO:scrape:Failure: Cannot create secret 2022-10-12T19:19:20.351 INFO:scrape:1 jobs: ['7063452'] 2022-10-12T19:19:20.352 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{centos_8}}'] 2022-10-12T19:19:20.352 INFO:scrape: 2022-10-12T19:19:20.353 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi035 with status 1: "BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a '!kafka_test,!amqp_test,!amqp_ssl_test,!modification_required,!manual_test'" 2022-10-12T19:19:20.353 INFO:scrape:1 jobs: ['7063457'] 2022-10-12T19:19:20.353 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'overrides', 'rgw/notifications/{beast', 'supported-all-distro$/{centos_8}', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}'] 2022-10-12T19:19:20.354 INFO:scrape: 2022-10-12T19:19:20.356 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 36 Fail ================================================================= [7063452] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:12:59 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063452/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063452/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f1f538d595e4e0d94c0e78cce5fed43 Cannot create secret [7063456] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 01:11:31 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063456/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063456/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1596ef02b99c44d29fe553e7c92d4ed3 rgw multisite test failures [7063457] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} ----------------------------------------------------------------- time: 00:29:22 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063457/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063457/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23c5880430a746d38b63f9b6b81dd1cf Command failed (bucket notification tests against different endpoints) on smithi035 with status 1: "BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/ rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/s rc/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a '!kafka_test,!amqp_test,!amqp_ssl_test,!modification_required,!manual_te st'" [7063460] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:19:29 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063460/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b9a0c77bfe0424fbff895ae6c7774b2 Command failed on smithi084 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" Pass ================================================================= [7063453] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:14:33 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063453/ [7063454] rgw/lifecycle/{cluster overrides supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests} time: 00:21:15 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063454/ [7063455] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:07:35 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063455/ [7063458] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:10:10 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063458/ [7063459] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}} time: 00:24:56 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063459/ [7063461] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:12:15 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063461/ [7063462] rgw/tools/{centos_latest cluster tasks} time: 02:43:08 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063462/ [7063463] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:26:03 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063463/ [7063464] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:15:37 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063464/ [7063465] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:10:39 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063465/ [7063466] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:14:12 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063466/ [7063467] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:06:32 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063467/ [7063468] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:11:28 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063468/ [7063469] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:20:20 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063469/ [7063470] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:53:57 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063470/ [7063471] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:13:20 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063471/ [7063472] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:11:54 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063472/ [7063473] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:11:35 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063473/ [7063474] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:09:58 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063474/ [7063475] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:10:47 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063475/ [7063476] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:13:04 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063476/ [7063477] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:07:01 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063477/ [7063478] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} time: 00:30:08 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063478/ [7063479] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:11:39 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063479/ [7063480] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:11:29 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063480/ [7063481] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:26:00 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063481/ [7063482] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:10:40 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063482/ [7063483] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:12:23 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063483/ [7063484] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:08:29 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063484/ [7063485] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:08:57 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063485/ [7063486] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:15:25 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063486/ [7063487] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:54:15 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063487/ [7063488] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:11:06 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063488/ [7063489] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:07:25 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063489/ [7063490] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:11:43 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063490/ [7063491] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:20 info: http://pulpito.ceph.com/yuriw-2022-10-12_16:19:39-rgw-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063491/