2022-03-28T02:21:32.918 INFO:root:teuthology version: 1.1.0-70390753 2022-03-28T02:21:32.920 INFO:teuthology.results:subset: '111/120000' 2022-03-28T02:21:32.920 INFO:teuthology.results:seed: '726' 2022-03-28T02:48:35.730 INFO:root:teuthology version: 1.1.0-70390753 2022-03-28T02:48:35.761 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-28T05:32:48.240 INFO:teuthology.results:Tests finished! gathering results... 2022-03-28T05:32:51.599 INFO:scrape:Found 40 jobs 2022-03-28T05:32:52.941 INFO:scrape:Found 5 distinct failure reasons 2022-03-28T05:32:52.942 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi139 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a lifecycle' 2022-03-28T05:32:52.943 INFO:scrape:1 jobs: ['6764101'] 2022-03-28T05:32:52.943 INFO:scrape:suites: ['overrides', 'rgw/lifecycle/{cluster', 'supported-random-distro$/{rhel_8}', 'tasks/rgw_s3tests}'] 2022-03-28T05:32:52.944 INFO:scrape: 2022-03-28T05:32:52.944 INFO:scrape:Failure: rgw multisite test failures 2022-03-28T05:32:52.945 INFO:scrape:1 jobs: ['6764108'] 2022-03-28T05:32:52.945 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2022-03-28T05:32:52.946 INFO:scrape: 2022-03-28T05:32:52.946 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi142 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test' 2022-03-28T05:32:52.947 INFO:scrape:1 jobs: ['6764111'] 2022-03-28T05:32:52.947 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'overrides', 'rgw/notifications/{beast', 'supported-all-distro$/{centos_8}', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}'] 2022-03-28T05:32:52.948 INFO:scrape: 2022-03-28T05:32:52.948 INFO:scrape:Failure: Command failed on smithi033 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2022-03-28T05:32:52.949 INFO:scrape:1 jobs: ['6764120'] 2022-03-28T05:32:52.949 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-03-28T05:32:52.950 INFO:scrape: 2022-03-28T05:32:52.950 INFO:scrape:Failure: Cannot create secret 2022-03-28T05:32:52.951 INFO:scrape:1 jobs: ['6764096'] 2022-03-28T05:32:52.951 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{rhel_8}}'] 2022-03-28T05:32:52.952 INFO:scrape: 2022-03-28T05:32:52.954 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 35 Fail ================================================================= [6764096] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:28:48 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764096/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764096/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8c61e95954ea457b94267f738b6151e5 Cannot create secret [6764101] rgw/lifecycle/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests} ----------------------------------------------------------------- time: 00:25:49 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764101/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764101/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5784019a2fb44a8b32bff4fd8513898 Command failed (s3 tests against rgw) on smithi139 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a lifecycle' [6764108] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:56:00 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764108/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764108/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f375ab9aefaa44c0be55f3c3626d8039 rgw multisite test failures [6764111] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} ----------------------------------------------------------------- time: 00:26:00 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764111/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764111/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7efb4ba0feb940fba0fac31a2d30a717 Command failed (bucket notification tests against different endpoints) on smithi142 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/ rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/s rc/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test' [6764120] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:21:29 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764120/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764120/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dbb380eb952b407287f556b923bff678 Command failed on smithi033 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" Pass ================================================================= [6764098] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:14:47 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764098/ [6764105] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:08:20 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764105/ [6764114] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:19:27 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764114/ [6764117] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}} time: 00:25:48 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764117/ [6764122] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:12:16 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764122/ [6764124] rgw/tools/{centos_latest cluster tasks} time: 02:50:56 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764124/ [6764126] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:27:21 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764126/ [6764129] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:16:02 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764129/ [6764131] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:48 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764131/ [6764133] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:24:23 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764133/ [6764135] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:07:36 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764135/ [6764138] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:11:42 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764138/ [6764140] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:20:09 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764140/ [6764142] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:46:32 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764142/ [6764145] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:21:16 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764145/ [6764147] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:12:37 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764147/ [6764149] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:09:43 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764149/ [6764151] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:20 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764151/ [6764154] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:13:55 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764154/ [6764156] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:13:27 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764156/ [6764158] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:07:29 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764158/ [6764160] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} time: 00:36:50 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764160/ [6764162] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:18:53 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764162/ [6764165] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:11:43 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764165/ [6764166] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:25:54 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764166/ [6764167] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:24 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764167/ [6764168] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:11:27 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764168/ [6764169] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:08:58 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764169/ [6764171] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:10:02 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764171/ [6764173] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:15:07 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764173/ [6764175] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:50:03 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764175/ [6764178] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:14:18 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764178/ [6764180] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:07:39 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764180/ [6764182] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:18:50 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764182/ [6764184] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:22 info: http://pulpito.ceph.com/yuriw-2022-03-27_21:52:16-rgw-quincy-distro-default-smithi/6764184/