2021-10-26T18:21:23.036 INFO:root:teuthology version: 1.1.0-c56135d1 2021-10-26T18:21:23.038 INFO:teuthology.results:subset: '1/20' 2021-10-26T18:21:23.038 INFO:teuthology.results:seed: '831' 2021-10-26T18:21:36.309 INFO:root:teuthology version: 1.1.0-c56135d1 2021-10-26T18:21:36.310 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-10-27T06:22:25.239 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-10-27T06:22:28.515 INFO:scrape:Found 35 jobs 2021-10-27T06:22:31.217 INFO:scrape:Found 5 distinct failure reasons 2021-10-27T06:22:31.218 INFO:scrape:Failure: "2021-10-26T18:41:30.999613+0000 mon.a (mon.0) 122 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log 2021-10-27T06:22:31.218 INFO:scrape:1 jobs: ['6462456'] 2021-10-27T06:22:31.219 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'objectstore/filestore-xfs', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/ec', 'tasks/rgw_ragweed', 'ubuntu_latest}'] 2021-10-27T06:22:31.219 INFO:scrape: 2021-10-27T06:22:31.219 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi041 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2021-10-27T06:22:31.220 INFO:scrape:4 jobs: ['6462463', '6462449', '6462474', '6462470'] 2021-10-27T06:22:31.220 INFO:scrape:suites intersection: ['ignore-pg-availability'] 2021-10-27T06:22:31.221 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/vault_kv', '2-kms/vault_old', '2-kms/vault_transit', '3-rgw/rgw', '4-tests/{s3tests}', 'first', 'ignore-pg-availability', 'objectstore', 'overrides', 'pool-type', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/sts/{cluster', 'rgw_frontend/beast', 'ststests}}', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{centos_8.stream}}', 'supported-random-distro$/{rhel_8}}', 'tasks/{0-install'] 2021-10-27T06:22:31.221 INFO:scrape: 2021-10-27T06:22:31.222 INFO:scrape:Failure: rgw multisite test failures 2021-10-27T06:22:31.222 INFO:scrape:2 jobs: ['6462466', '6462446'] 2021-10-27T06:22:31.223 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2021-10-27T06:22:31.223 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2021-10-27T06:22:31.224 INFO:scrape: 2021-10-27T06:22:31.224 INFO:scrape:Failure: Command failed on smithi042 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2021-10-27T06:22:31.224 INFO:scrape:2 jobs: ['6462447', '6462450'] 2021-10-27T06:22:31.225 INFO:scrape:suites intersection: ['overrides'] 2021-10-27T06:22:31.226 INFO:scrape:suites union: ['bluestore-bitmap', 'fixed-2', 'frontend/beast', 'overrides', 'rgw/notifications/{beast', 'rgw/tempest/{clusters/fixed-1', 'supported-all-distro$/{centos_8}', 'tasks/rgw_tempest', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}', 'ubuntu_latest}'] 2021-10-27T06:22:31.226 INFO:scrape: 2021-10-27T06:22:31.227 INFO:scrape:Dead: b'2021-10-26T19:10:57.176 INFO:tasks.rgw.client.0.smithi058.stdout:' 2021-10-27T06:22:31.227 INFO:scrape:1 jobs: ['6462455'] 2021-10-27T06:22:31.227 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 'supported-random-distro$/{centos_8}}'] 2021-10-27T06:22:31.228 INFO:scrape: 2021-10-27T06:22:31.229 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/ failed: 9 dead: 0 running: 1 waiting: 0 queued: 0 passed: 25 Fail ================================================================= [6462446] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 04:41:08 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462446/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462446/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d58e24870c924a1faaf00132efb9335d rgw multisite test failures [6462447] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} ----------------------------------------------------------------- time: 00:32:48 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462447/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462447/ Command failed on smithi042 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' [6462449] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8.stream} tasks/{0-install first ststests}} ----------------------------------------------------------------- time: 00:31:18 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462449/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462449/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bec9769381914964802923016e8ce448 Command failed (s3 tests against rgw) on smithi027 with status 1: 'S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a test_of_sts' [6462450] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:18:30 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462450/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462450/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0433097fa37e451c825fecb9544586c5 Command failed on smithi186 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [6462456] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} ----------------------------------------------------------------- time: 00:09:33 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462456/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462456/ "2021-10-26T18:41:30.999613+0000 mon.a (mon.0) 122 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [6462463] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8.stream}} ----------------------------------------------------------------- time: 00:15:44 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462463/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462463/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e627bd3d0aa848949a097c7851d279dc Command failed (s3 tests against rgw) on smithi041 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6462466] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8.stream} tasks/test_multi} ----------------------------------------------------------------- time: 00:43:51 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462466/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462466/ sentry: https://sentry.ceph.com/organizations/ceph/?query=592506e30936430bbd7a5e4be1e61834 rgw multisite test failures [6462470] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:27:02 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462470/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462470/ sentry: https://sentry.ceph.com/organizations/ceph/?query=971f27c9914a4fa99d9efa8094bd1c87 Command failed (s3 tests against rgw) on smithi026 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" [6462474] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:26:31 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462474/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462474/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4964cc1863ca4d92a797ffaa6da8cacc Command failed (s3 tests against rgw) on smithi166 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,! fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" Running ================================================================= [6462455] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462455/ Pass ================================================================= [6462442] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:15 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462442/ [6462443] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:21:21 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462443/ [6462444] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:16:59 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462444/ [6462445] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:39 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462445/ [6462448] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:13:30 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462448/ [6462451] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:15:21 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462451/ [6462452] rgw/tools/{centos_latest cluster tasks} time: 02:51:58 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462452/ [6462453] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:17:51 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462453/ [6462454] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:30 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462454/ [6462457] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:13:36 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462457/ [6462458] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:16:41 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462458/ [6462459] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:11:43 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462459/ [6462460] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:15:03 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462460/ [6462461] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} time: 00:15:03 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462461/ [6462462] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:20 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462462/ [6462464] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:16:03 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462464/ [6462465] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:11:08 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462465/ [6462467] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:13:26 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462467/ [6462468] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:03 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462468/ [6462469] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:47 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462469/ [6462471] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:12:15 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462471/ [6462472] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:24:51 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462472/ [6462473] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:46 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462473/ [6462475] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:09:59 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462475/ [6462476] rgw/singleton/{all/radosgw-admin frontend/beast objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8.stream}} time: 00:14:40 info: http://pulpito.ceph.com/ivancich-2021-10-26_18:20:44-rgw-wip-eric-testing-1-distro-basic-smithi/6462476/