2023-03-24T13:25:44.829 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-24T13:25:44.830 INFO:teuthology.results:subset: '111/120000' 2023-03-24T13:25:44.831 INFO:teuthology.results:seed: '5553' 2023-03-24T13:49:34.888 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-24T13:49:34.892 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-24T16:26:48.260 INFO:teuthology.results:Tests finished! gathering results... 2023-03-24T16:26:51.626 INFO:scrape:Found 41 jobs 2023-03-24T16:26:52.848 INFO:scrape:Found 3 distinct failure reasons 2023-03-24T16:26:52.849 INFO:scrape:Failure: Command failed on smithi107 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' 2023-03-24T16:26:52.850 INFO:scrape:1 jobs: ['7218958'] 2023-03-24T16:26:52.850 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-03-24T16:26:52.850 INFO:scrape: 2023-03-24T16:26:52.851 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=23eb3b2f0fc65087846571af4e15146a980fc03d 2023-03-24T16:26:52.851 INFO:scrape:2 jobs: ['7218951', '7218974'] 2023-03-24T16:26:52.852 INFO:scrape:suites intersection: ['overrides', 'rgw/hadoop-s3a/{clusters/fixed-2', 's3a-hadoop}'] 2023-03-24T16:26:52.852 INFO:scrape:suites union: ['hadoop/default', 'hadoop/v32', 'overrides', 'rgw/hadoop-s3a/{clusters/fixed-2', 's3a-hadoop}'] 2023-03-24T16:26:52.853 INFO:scrape: 2023-03-24T16:26:52.853 INFO:scrape:Failure: rgw multisite test failures 2023-03-24T16:26:52.854 INFO:scrape:1 jobs: ['7218954'] 2023-03-24T16:26:52.854 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2023-03-24T16:26:52.855 INFO:scrape: 2023-03-24T16:26:52.858 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 37 Fail ================================================================= [7218951] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} ----------------------------------------------------------------- time: 00:07:44 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218951/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218951/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9807653cd75f4c42b14c1e2676581d27 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =23eb3b2f0fc65087846571af4e15146a980fc03d [7218954] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 00:40:39 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218954/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218954/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66e0b3e3a87f4cffa333263b5093983c rgw multisite test failures [7218958] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:19:58 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218958/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218958/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a770e46d6d2e431991007493c1f78b85 Command failed on smithi107 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7218974] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} ----------------------------------------------------------------- time: 00:07:46 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218974/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218974/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b33e49989f6d4bbaa00c2a38d9f72f50 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =23eb3b2f0fc65087846571af4e15146a980fc03d Pass ================================================================= [7218949] rgw/cloud-transition/{cluster overrides supported-random-distro$/{ubuntu_latest} tasks/cloud_transition_s3tests} time: 00:30:14 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218949/ [7218950] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:23:20 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218950/ [7218952] rgw/lifecycle/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests} time: 00:30:07 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218952/ [7218953] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:53 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218953/ [7218955] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:33:10 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218955/ [7218956] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:14:40 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218956/ [7218957] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:31:25 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218957/ [7218959] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:15:03 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218959/ [7218960] rgw/tools/{centos_latest cluster tasks} time: 02:47:09 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218960/ [7218961] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:30:47 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218961/ [7218962] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:18:59 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218962/ [7218963] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:04 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218963/ [7218964] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:18:14 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218964/ [7218965] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:10:09 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218965/ [7218966] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:14:23 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218966/ [7218967] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:29:19 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218967/ [7218968] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:55:19 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218968/ [7218969] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:14:23 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218969/ [7218970] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:15:33 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218970/ [7218971] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:12:08 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218971/ [7218972] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:11 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218972/ [7218973] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:14:23 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218973/ [7218975] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:09:59 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218975/ [7218976] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} time: 00:32:14 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218976/ [7218977] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:14:59 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218977/ [7218978] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:48 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218978/ [7218979] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:28:36 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218979/ [7218980] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:50 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218980/ [7218981] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:16:46 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218981/ [7218982] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:11:46 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218982/ [7218983] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:11:45 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218983/ [7218984] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:26 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218984/ [7218985] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:00:30 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218985/ [7218986] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:17:06 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218986/ [7218987] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:04 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218987/ [7218988] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:15:18 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218988/ [7218989] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:14:18 info: http://pulpito.ceph.com/yuriw-2023-03-24_13:24:31-rgw-quincy-release-distro-default-smithi/7218989/