2022-02-12T02:09:59.119 INFO:root:teuthology version: 1.1.0-7cebb3f2 2022-02-12T02:09:59.121 INFO:teuthology.results:subset: '111/120000' 2022-02-12T02:09:59.121 INFO:teuthology.results:seed: '5421' 2022-02-12T02:31:23.690 INFO:root:teuthology version: 1.1.0-7cebb3f2 2022-02-12T02:31:23.731 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-12T07:37:47.297 INFO:teuthology.results:Tests finished! gathering results... 2022-02-12T07:37:51.180 INFO:scrape:Found 39 jobs 2022-02-12T07:37:51.319 WARNING:scrape:Not-understood crash indication 2022-02-12T02:52:01.903 INFO:tasks.rgw.c1.client.0.smithi123.stderr:daemon-helper: command crashed with signal 11 2022-02-12T07:37:51.320 WARNING:scrape:Not-understood crash indication 2022-02-12T07:37:51.339 WARNING:scrape:Not-understood crash indication 2022-02-12T02:52:01.903 INFO:tasks.rgw.c1.client.0.smithi123.stderr:daemon-helper: command crashed with signal 11 2022-02-12T07:37:51.340 WARNING:scrape:Not-understood crash indication 2022-02-12T07:37:51.357 WARNING:scrape:Not-understood crash indication 2022-02-12T02:52:01.903 INFO:tasks.rgw.c1.client.0.smithi123.stderr:daemon-helper: command crashed with signal 11 2022-02-12T07:37:51.358 WARNING:scrape:Not-understood crash indication 2022-02-12T07:37:52.190 INFO:scrape:Found 3 distinct failure reasons 2022-02-12T07:37:52.192 INFO:scrape:Failure: rgw multisite test failures 2022-02-12T07:37:52.192 INFO:scrape:2 jobs: ['6677697', '6677675'] 2022-02-12T07:37:52.192 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2022-02-12T07:37:52.193 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2022-02-12T07:37:52.194 INFO:scrape: 2022-02-12T07:37:52.194 INFO:scrape:Failure: Command failed on smithi173 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2022-02-12T07:37:52.196 INFO:scrape:1 jobs: ['6677679'] 2022-02-12T07:37:52.196 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-02-12T07:37:52.197 INFO:scrape: 2022-02-12T07:37:52.197 INFO:scrape:Failure: Cannot create secret 2022-02-12T07:37:52.198 INFO:scrape:1 jobs: ['6677672'] 2022-02-12T07:37:52.198 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{centos_8}}'] 2022-02-12T07:37:52.199 INFO:scrape: 2022-02-12T07:37:52.201 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 35 Fail ================================================================= [6677672] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:13:09 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677672/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677672/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c978b8ac8ec548afae7548cfe691d5f7 Cannot create secret [6677675] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:44:57 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677675/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677675/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce64ae5487d74e998f5502c1e3cf380e rgw multisite test failures [6677679] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:19:48 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677679/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677679/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d9c78dfdfa14c0589da00858ded0ed7 Command failed on smithi173 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" [6677697] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 05:08:39 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677697/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677697/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8afa9457f8a4e61ba2686e997658143 rgw multisite test failures Pass ================================================================= [6677673] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:14:21 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677673/ [6677674] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:07:48 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677674/ [6677676] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:29:11 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677676/ [6677677] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:23:33 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677677/ [6677678] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}} time: 00:25:04 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677678/ [6677680] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:11:32 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677680/ [6677681] rgw/tools/{centos_latest cluster tasks} time: 02:42:43 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677681/ [6677682] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:25:52 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677682/ [6677683] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:16:10 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677683/ [6677684] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:10:53 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677684/ [6677685] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:12:37 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677685/ [6677686] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:06:53 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677686/ [6677687] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:10:11 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677687/ [6677688] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:53 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677688/ [6677689] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:47:11 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677689/ [6677690] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:11:51 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677690/ [6677691] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:12:36 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677691/ [6677692] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:09:01 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677692/ [6677693] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:10:10 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677693/ [6677694] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:11:52 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677694/ [6677695] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:12:58 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677695/ [6677696] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:07:24 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677696/ [6677698] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:23:17 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677698/ [6677699] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:11:11 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677699/ [6677700] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:26:00 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677700/ [6677701] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:10:38 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677701/ [6677702] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:25:10 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677702/ [6677703] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:08:31 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677703/ [6677704] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:09:12 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677704/ [6677705] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:15:03 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677705/ [6677706] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:45:05 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677706/ [6677707] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:12:12 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677707/ [6677708] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:07:16 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677708/ [6677709] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:10:04 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677709/ [6677710] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:02 info: http://pulpito.ceph.com/yuriw-2022-02-11_19:48:08-rgw-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677710/