2023-02-18T08:26:46.424 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5 2023-02-18T08:26:46.425 INFO:teuthology.results:subset: '1/20' 2023-02-18T08:26:46.425 INFO:teuthology.results:seed: '831' 2023-02-18T08:58:38.410 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5 2023-02-18T08:58:38.434 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-02-18T11:33:50.310 INFO:teuthology.results:Tests finished! gathering results... 2023-02-18T11:33:53.071 INFO:scrape:Found 39 jobs 2023-02-18T11:33:56.298 INFO:scrape:Found 3 distinct failure reasons 2023-02-18T11:33:56.299 INFO:scrape:Failure: rgw multisite test failures 2023-02-18T11:33:56.299 INFO:scrape:2 jobs: ['7179699', '7179720'] 2023-02-18T11:33:56.300 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2023-02-18T11:33:56.300 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2023-02-18T11:33:56.300 INFO:scrape: 2023-02-18T11:33:56.301 INFO:scrape:Failure: Command failed on smithi133 with status 1: 'cd /home/ubuntu/.aws/models/s3/2006-03-01/ && cp /home/ubuntu/cephtest/ceph/examples/boto3/service-2.sdk-extras.json service-2.sdk-extras.json' 2023-02-18T11:33:56.301 INFO:scrape:2 jobs: ['7179700', '7179704'] 2023-02-18T11:33:56.301 INFO:scrape:suites intersection: ['overrides'] 2023-02-18T11:33:56.301 INFO:scrape:suites union: ['bluestore-bitmap', 'fixed-2', 'frontend/beast', 'overrides', 'rgw/notifications/{beast', 'rgw/tempest/{clusters/fixed-1', 'supported-all-distro$/{centos_8}', 'tasks/rgw_tempest', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}', 'ubuntu_latest}'] 2023-02-18T11:33:56.302 INFO:scrape: 2023-02-18T11:33:56.302 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi176 with status 1: 'source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m lifecycle' 2023-02-18T11:33:56.302 INFO:scrape:2 jobs: ['7179697', '7179695'] 2023-02-18T11:33:56.303 INFO:scrape:suites intersection: ['overrides', 's3tests-branch', 'tasks/rgw_s3tests}'] 2023-02-18T11:33:56.303 INFO:scrape:suites union: ['overrides', 'rgw/dbstore/{cluster', 'rgw/lifecycle/{cluster', 's3tests-branch', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rgw_s3tests}'] 2023-02-18T11:33:56.303 INFO:scrape: 2023-02-18T11:33:56.305 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 33 Fail ================================================================= [7179695] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests} ----------------------------------------------------------------- time: 00:19:37 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179695/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179695/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a6bc9e4b4a064333b470d77d7861cb47 Command failed (s3 tests against rgw) on smithi008 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not fails_on_dbstore'" [7179697] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests} ----------------------------------------------------------------- time: 00:23:56 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179697/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179697/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4336c5d446df4dce8184dc6de271a198 Command failed (s3 tests against rgw) on smithi176 with status 1: 'source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m lifecycle' [7179699] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 02:31:43 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179699/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179699/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6120ed0fab334fe79ee48e316bb7042b rgw multisite test failures [7179700] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} ----------------------------------------------------------------- time: 00:13:13 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179700/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179700/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a7e46f2c8694eb982f90846339e273d Command failed on smithi133 with status 1: 'cd /home/ubuntu/.aws/models/s3/2006-03-01/ && cp /home/ubuntu/cephtest/ceph/examples/boto3/service-2.sdk-extras.json service-2.sdk-extras.json' [7179704] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:20:18 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179704/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d639e366a90649ecaa5f0f6e7a106e70 Command failed on smithi164 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7179720] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 02:13:42 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179720/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179720/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02137ad8846d4585ba8ace2d70949fcc rgw multisite test failures Pass ================================================================= [7179692] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:54 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179692/ [7179693] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests} time: 00:32:23 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179693/ [7179694] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} time: 00:24:52 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179694/ [7179696] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:18:05 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179696/ [7179698] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:24 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179698/ [7179701] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest} time: 00:14:23 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179701/ [7179702] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:14:10 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179702/ [7179703] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:30:19 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179703/ [7179705] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:15:23 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179705/ [7179706] rgw/tools/{centos_latest cluster tasks} time: 02:47:07 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179706/ [7179707] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest} time: 00:18:12 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179707/ [7179708] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:12:39 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179708/ [7179709] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} time: 00:18:12 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179709/ [7179710] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:09:43 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179710/ [7179711] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:15:15 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179711/ [7179712] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:37 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179712/ [7179713] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} time: 00:16:43 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179713/ [7179714] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:14:58 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179714/ [7179715] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:11:30 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179715/ [7179716] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:32 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179716/ [7179717] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}} time: 00:14:20 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179717/ [7179718] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:17:28 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179718/ [7179719] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:09:37 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179719/ [7179721] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:15:10 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179721/ [7179722] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:11 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179722/ [7179723] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:43 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179723/ [7179724] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}} time: 00:14:16 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179724/ [7179725] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:11:32 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179725/ [7179726] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:11:42 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179726/ [7179727] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:59 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179727/ [7179728] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} time: 00:16:21 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179728/ [7179729] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:10 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179729/ [7179730] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:15:08 info: http://pulpito.ceph.com/ivancich-2023-02-17_21:47:17-rgw-wip-eric-testing-1-distro-default-smithi/7179730/