2021-10-21T18:11:32.876 INFO:root:teuthology version: 1.1.0-c56135d1 2021-10-21T18:11:32.878 INFO:teuthology.results:subset: '1/20' 2021-10-21T18:11:32.878 INFO:teuthology.results:seed: '831' 2021-10-21T18:20:46.382 INFO:root:teuthology version: 1.1.0-c56135d1 2021-10-21T18:20:46.415 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-10-22T06:21:42.007 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-10-22T06:21:45.129 INFO:scrape:Found 35 jobs 2021-10-22T06:21:47.113 INFO:scrape:Found 5 distinct failure reasons 2021-10-22T06:21:47.114 INFO:scrape:Failure: rgw multisite test failures 2021-10-22T06:21:47.114 INFO:scrape:1 jobs: ['6456206'] 2021-10-22T06:21:47.115 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2021-10-22T06:21:47.115 INFO:scrape: 2021-10-22T06:21:47.116 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=db9fd61ac97586374adaaf307fb4cd1a34a0e1e6 2021-10-22T06:21:47.116 INFO:scrape:1 jobs: ['6456230'] 2021-10-22T06:21:47.117 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/vault_old', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{rhel_8}}'] 2021-10-22T06:21:47.117 INFO:scrape: 2021-10-22T06:21:47.117 INFO:scrape:Failure: Command failed on smithi012 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' 2021-10-22T06:21:47.118 INFO:scrape:1 jobs: ['6456210'] 2021-10-22T06:21:47.118 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2021-10-22T06:21:47.119 INFO:scrape: 2021-10-22T06:21:47.119 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi116 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a test_of_sts' 2021-10-22T06:21:47.120 INFO:scrape:3 jobs: ['6456209', '6456223', '6456234'] 2021-10-22T06:21:47.120 INFO:scrape:suites intersection: ['ignore-pg-availability'] 2021-10-22T06:21:47.120 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/vault_kv', '2-kms/vault_transit', '3-rgw/rgw', '4-tests/{s3tests}', 'first', 'ignore-pg-availability', 'objectstore', 'overrides', 'pool-type', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/sts/{cluster', 'rgw_frontend/beast', 'ststests}}', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{centos_8.stream}}', 'supported-random-distro$/{rhel_8}}', 'tasks/{0-install'] 2021-10-22T06:21:47.121 INFO:scrape: 2021-10-22T06:21:47.121 INFO:scrape:Dead: b'2021-10-22T06:21:26.484 DEBUG:teuthology.exit:Finished running handlers' 2021-10-22T06:21:47.122 INFO:scrape:1 jobs: ['6456215'] 2021-10-22T06:21:47.122 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{centos_8}}'] 2021-10-22T06:21:47.123 INFO:scrape: 2021-10-22T06:21:47.124 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/ failed: 6 dead: 0 running: 1 waiting: 0 queued: 0 passed: 28 Fail ================================================================= [6456206] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 05:26:36 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456206/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456206/ sentry: https://sentry.ceph.com/organizations/ceph/?query=32e54d3585b14c62aba5ca740a53ef37 rgw multisite test failures [6456209] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8.stream} tasks/{0-install first ststests}} ----------------------------------------------------------------- time: 00:29:38 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456209/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ca3dbb1207541b8beee346e532f1b92 Command failed (s3 tests against rgw) on smithi116 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a test_of_sts' [6456210] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:18:17 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456210/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456210/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bf6ebfab625d435a8489a10ac5ac364e Command failed on smithi012 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [6456223] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8.stream}} ----------------------------------------------------------------- time: 00:15:14 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456223/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456223/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9319de9b13b449397fcc362af5ca1ce Command failed (s3 tests against rgw) on smithi042 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6456230] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:18:00 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456230/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456230/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79b051af87e24e2fbdabb4b64aac76d4 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=db9 fd61ac97586374adaaf307fb4cd1a34a0e1e6 [6456234] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:25:48 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456234/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456234/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e032ed5ea3744ecfb441573823aa2869 Command failed (s3 tests against rgw) on smithi121 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" Running ================================================================= [6456215] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456215/ Pass ================================================================= [6456202] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:59 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456202/ [6456203] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:20:00 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456203/ [6456204] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:15:52 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456204/ [6456205] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:08:38 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456205/ [6456207] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:30:15 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456207/ [6456208] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:12:05 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456208/ [6456211] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:13:37 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456211/ [6456212] rgw/tools/{centos_latest cluster tasks} time: 02:44:07 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456212/ [6456213] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:17:05 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456213/ [6456214] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:32 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456214/ [6456216] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:08:25 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456216/ [6456217] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:12:08 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456217/ [6456218] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:00 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456218/ [6456219] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:11:05 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456219/ [6456220] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:13:24 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456220/ [6456221] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} time: 00:13:13 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456221/ [6456222] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:11:53 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456222/ [6456224] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:14:23 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456224/ [6456225] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:08:45 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456225/ [6456226] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8.stream} tasks/test_multi} time: 00:41:24 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456226/ [6456227] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:12:47 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456227/ [6456228] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:13:32 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456228/ [6456229] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:11 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456229/ [6456231] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:09:19 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456231/ [6456232] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:25:00 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456232/ [6456233] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:15:36 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456233/ [6456235] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:08:35 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456235/ [6456236] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} time: 00:13:29 info: http://pulpito.ceph.com/ivancich-2021-10-21_17:31:10-rgw-wip-eric-testing-2-distro-basic-smithi/6456236/