2021-11-17T04:02:07.360 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-17T04:02:07.361 INFO:teuthology.results:subset: '1/20' 2021-11-17T04:02:07.361 INFO:teuthology.results:seed: '831' 2021-11-17T04:34:18.538 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-17T04:34:18.590 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-11-17T14:59:07.031 INFO:teuthology.results:Tests finished! gathering results... 2021-11-17T14:59:10.527 INFO:scrape:Found 35 jobs 2021-11-17T14:59:10.693 WARNING:scrape:Not-understood crash indication 2021-11-17T04:45:29.966 INFO:tasks.rgw.c1.client.0.smithi064.stderr:daemon-helper: command crashed with signal 11 2021-11-17T14:59:10.694 WARNING:scrape:Not-understood crash indication 2021-11-17T14:59:12.194 INFO:scrape:Found 5 distinct failure reasons 2021-11-17T14:59:12.195 INFO:scrape:Dead: b'2021-11-17T14:56:40.360 DEBUG:teuthology.exit:Finished running handlers' 2021-11-17T14:59:12.196 INFO:scrape:2 jobs: ['6508611', '6508631'] 2021-11-17T14:59:12.196 INFO:scrape:suites intersection: ['ignore-pg-availability'] 2021-11-17T14:59:12.197 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2021-11-17T14:59:12.197 INFO:scrape: 2021-11-17T14:59:12.197 INFO:scrape:Failure: "2021-11-17T04:49:48.607575+0000 mon.a (mon.0) 202 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log 2021-11-17T14:59:12.198 INFO:scrape:1 jobs: ['6508667'] 2021-11-17T14:59:12.198 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/ec', 'tasks/rgw_bucket_quota', 'ubuntu_latest}'] 2021-11-17T14:59:12.199 INFO:scrape: 2021-11-17T14:59:12.199 INFO:scrape:Failure: Command failed on smithi135 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2021-11-17T14:59:12.200 INFO:scrape:1 jobs: ['6508620'] 2021-11-17T14:59:12.200 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2021-11-17T14:59:12.201 INFO:scrape: 2021-11-17T14:59:12.201 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi043 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a test_of_sts' 2021-11-17T14:59:12.201 INFO:scrape:1 jobs: ['6508618'] 2021-11-17T14:59:12.202 INFO:scrape:suites: ['first', 'ignore-pg-availability', 'objectstore', 'overrides', 'pool-type', 'rgw/sts/{cluster', 'rgw_frontend/beast', 'ststests}}', 'supported-random-distro$/{centos_8.stream}', 'tasks/{0-install'] 2021-11-17T14:59:12.202 INFO:scrape: 2021-11-17T14:59:12.203 INFO:scrape:Failure: rgw multisite test failures 2021-11-17T14:59:12.203 INFO:scrape:1 jobs: ['6508656'] 2021-11-17T14:59:12.204 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8.stream}', 'tasks/test_multi}'] 2021-11-17T14:59:12.204 INFO:scrape: 2021-11-17T14:59:12.206 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/ failed: 4 dead: 2 running: 0 waiting: 0 queued: 0 passed: 29 Fail ================================================================= [6508618] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8.stream} tasks/{0-install first ststests}} ----------------------------------------------------------------- time: 00:29:35 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508618/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508618/ sentry: https://sentry.ceph.com/organizations/ceph/?query=719decd84f264053a262c03f486e7418 Command failed (s3 tests against rgw) on smithi043 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a test_of_sts' [6508620] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:19:43 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508620/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508620/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1768e6ddc0354c918b1da0d3108ad799 Command failed on smithi135 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" [6508656] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8.stream} tasks/test_multi} ----------------------------------------------------------------- time: 01:42:57 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508656/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508656/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8315ba1d4719465d82bd2d4608c4bead rgw multisite test failures [6508667] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} ----------------------------------------------------------------- time: 00:08:56 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508667/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508667/ "2021-11-17T04:49:48.607575+0000 mon.a (mon.0) 202 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log Dead ================================================================= [6508611] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508611/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508611/ [6508631] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508631/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508631/ Pass ================================================================= [6508603] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:58 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508603/ [6508605] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:20:09 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508605/ [6508607] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:14:58 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508607/ [6508609] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:08:38 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508609/ [6508614] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:30:24 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508614/ [6508616] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:12:06 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508616/ [6508623] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:13:40 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508623/ [6508625] rgw/tools/{centos_latest cluster tasks} time: 02:43:35 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508625/ [6508627] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:16:01 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508627/ [6508629] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:10:59 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508629/ [6508634] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:07:36 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508634/ [6508636] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:12:03 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508636/ [6508638] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:26 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508638/ [6508640] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:11:38 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508640/ [6508642] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:12:46 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508642/ [6508645] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} time: 00:13:15 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508645/ [6508647] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:10:40 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508647/ [6508649] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8.stream}} time: 00:14:53 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508649/ [6508651] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:13:36 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508651/ [6508653] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:07:53 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508653/ [6508658] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:16:27 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508658/ [6508660] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:11:54 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508660/ [6508662] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:11:37 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508662/ [6508664] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:26:46 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508664/ [6508669] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:25:35 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508669/ [6508671] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:15:43 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508671/ [6508673] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:26:56 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508673/ [6508675] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:08:04 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508675/ [6508678] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} time: 00:13:23 info: http://pulpito.ceph.com/ivancich-2021-11-17_03:32:43-rgw-wip-eric-testing-1-distro-basic-smithi/6508678/