2021-11-16T11:32:04.265 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-16T11:32:04.266 INFO:teuthology.results:subset: '1/20' 2021-11-16T11:32:04.266 INFO:teuthology.results:seed: '831' 2021-11-16T12:12:17.492 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-16T12:12:17.524 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-11-17T00:03:11.560 INFO:teuthology.results:Tests finished! gathering results... 2021-11-17T00:03:15.580 INFO:scrape:Found 35 jobs 2021-11-17T00:03:15.924 WARNING:scrape:Not-understood crash indication 2021-11-16T12:17:40.963 INFO:tasks.rgw.c1.client.0.smithi023.stderr:daemon-helper: command crashed with signal 11 2021-11-17T00:03:15.925 WARNING:scrape:Not-understood crash indication 2021-11-17T00:03:17.629 INFO:scrape:Found 4 distinct failure reasons 2021-11-17T00:03:17.630 INFO:scrape:Dead: b'2021-11-16T23:44:37.097 DEBUG:teuthology.exit:Finished running handlers' 2021-11-17T00:03:17.631 INFO:scrape:2 jobs: ['6506045', '6506054'] 2021-11-17T00:03:17.632 INFO:scrape:suites intersection: ['ignore-pg-availability'] 2021-11-17T00:03:17.632 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2021-11-17T00:03:17.633 INFO:scrape: 2021-11-17T00:03:17.634 INFO:scrape:Failure: "2021-11-16T12:31:31.600608+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log 2021-11-17T00:03:17.635 INFO:scrape:2 jobs: ['6506075', '6506042'] 2021-11-17T00:03:17.635 INFO:scrape:suites intersection: [] 2021-11-17T00:03:17.636 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'frontend/beast', 'ignore-pg-availability', 'objectstore/filestore-xfs', 'overrides', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/singleton/{all/radosgw-admin', 'rgw_pool_type/replicated', 'supported-random-distro$/{centos_8.stream}}', 'supported-random-distro$/{centos_8}}'] 2021-11-17T00:03:17.637 INFO:scrape: 2021-11-17T00:03:17.637 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi007 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a test_of_sts' 2021-11-17T00:03:17.638 INFO:scrape:1 jobs: ['6506048'] 2021-11-17T00:03:17.639 INFO:scrape:suites: ['first', 'ignore-pg-availability', 'objectstore', 'overrides', 'pool-type', 'rgw/sts/{cluster', 'rgw_frontend/beast', 'ststests}}', 'supported-random-distro$/{centos_8.stream}', 'tasks/{0-install'] 2021-11-17T00:03:17.639 INFO:scrape: 2021-11-17T00:03:17.640 INFO:scrape:Failure: Command failed on smithi026 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds|.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding'" 2021-11-17T00:03:17.641 INFO:scrape:1 jobs: ['6506049'] 2021-11-17T00:03:17.641 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2021-11-17T00:03:17.642 INFO:scrape: 2021-11-17T00:03:17.644 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/ failed: 4 dead: 2 running: 0 waiting: 0 queued: 0 passed: 29 Fail ================================================================= [6506042] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:20:22 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506042/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506042/ "2021-11-16T11:59:13.201621+0000 mon.a (mon.0) 180 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [6506048] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8.stream} tasks/{0-install first ststests}} ----------------------------------------------------------------- time: 00:30:16 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506048/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506048/ sentry: https://sentry.ceph.com/organizations/ceph/?query=acd0840940ca4b60a5054c3fd1fd2e1b Command failed (s3 tests against rgw) on smithi007 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a test_of_sts' [6506049] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:21:15 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506049/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506049/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a23c742d97a4430d861ef0fd6f0e3dd1 Command failed on smithi026 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test _account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|. *test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_serv ices_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*tes t_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticw eb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.* test_object_services.PublicObjectTest.test_access_public_container_object_w ithout_using_creds|.*test_object_services.ObjectTest.test_create_object_wit h_transfer_encoding'" [6506075] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} ----------------------------------------------------------------- time: 00:13:37 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506075/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506075/ "2021-11-16T12:31:31.600608+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log Dead ================================================================= [6506045] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506045/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506045/ hit max job timeout [6506054] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506054/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506054/ hit max job timeout Pass ================================================================= [6506041] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:06 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506041/ [6506043] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:16:13 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506043/ [6506044] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:09:09 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506044/ [6506046] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:29:42 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506046/ [6506047] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:12:48 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506047/ [6506050] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:41 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506050/ [6506051] rgw/tools/{centos_latest cluster tasks} time: 02:44:53 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506051/ [6506052] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:16:21 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506052/ [6506053] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:54 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506053/ [6506055] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:08:36 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506055/ [6506056] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:12:02 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506056/ [6506057] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:12 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506057/ [6506058] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:11:50 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506058/ [6506059] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:12:50 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506059/ [6506060] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} time: 00:13:22 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506060/ [6506061] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:23 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506061/ [6506062] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8.stream}} time: 00:15:10 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506062/ [6506063] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:13:35 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506063/ [6506064] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:07:54 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506064/ [6506065] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8.stream} tasks/test_multi} time: 00:34:53 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506065/ [6506066] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:12:11 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506066/ [6506067] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:12:55 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506067/ [6506068] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:24 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506068/ [6506069] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:28:11 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506069/ [6506070] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:09:27 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506070/ [6506071] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:24:44 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506071/ [6506072] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:04 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506072/ [6506073] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:28:08 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506073/ [6506074] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:08:11 info: http://pulpito.ceph.com/ivancich-2021-11-16_04:34:19-rgw-wip-eric-testing-1-distro-basic-smithi/6506074/