2022-12-29T00:24:27.248 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-29T00:24:27.249 INFO:teuthology.results:subset: '111/120000' 2022-12-29T00:24:27.249 INFO:teuthology.results:seed: '2478' 2022-12-29T00:42:03.737 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-29T00:42:03.765 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-12-29T03:41:19.753 INFO:teuthology.results:Tests finished! gathering results... 2022-12-29T03:41:22.849 INFO:scrape:Found 43 jobs 2022-12-29T03:41:26.195 INFO:scrape:Found 3 distinct failure reasons 2022-12-29T03:41:26.196 INFO:scrape:Failure: rgw multisite, rgw multisite pubsub, test failures 2022-12-29T03:41:26.197 INFO:scrape:4 jobs: ['7128965', '7128975', '7128953', '7128986'] 2022-12-29T03:41:26.197 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-12-29T03:41:26.198 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-12-29T03:41:26.198 INFO:scrape: 2022-12-29T03:41:26.198 INFO:scrape:Failure: Cannot create secret 2022-12-29T03:41:26.199 INFO:scrape:1 jobs: ['7128950'] 2022-12-29T03:41:26.199 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability}', 'rgw/crypt/{0-cluster/fixed-1'] 2022-12-29T03:41:26.200 INFO:scrape: 2022-12-29T03:41:26.200 INFO:scrape:Failure: Command failed on smithi093 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' 2022-12-29T03:41:26.200 INFO:scrape:2 jobs: ['7128978', '7128956'] 2022-12-29T03:41:26.201 INFO:scrape:suites intersection: ['overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-12-29T03:41:26.201 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-12-29T03:41:26.201 INFO:scrape: 2022-12-29T03:41:26.203 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 36 Fail ================================================================= [7128950] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:16:18 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128950/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128950/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7973b35e92e4e3888ff4c7016ec714d Cannot create secret [7128953] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 00:42:45 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128953/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128953/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5758334868fc4bc7add82a6c3a2a50ef rgw multisite pubsub, test failures [7128956] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:22:17 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128956/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128956/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e072503c58e84cdca143f649fcda6ed1 Command failed on smithi033 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7128965] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} ----------------------------------------------------------------- time: 02:26:13 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128965/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128965/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae980b50b72043188283087dc3cc3549 rgw multisite, rgw multisite pubsub, test failures [7128975] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 02:57:47 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128975/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128975/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78320e33fe164683b54c95d453bedf33 rgw multisite, rgw multisite pubsub, test failures [7128978] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:22:36 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128978/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128978/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9924fcb862a440808fda531201825afa Command failed on smithi093 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7128986] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} ----------------------------------------------------------------- time: 02:38:45 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128986/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128986/ sentry: https://sentry.ceph.com/organizations/ceph/?query=daa2d5bec2764cb39415256e50e5509b rgw multisite, rgw multisite pubsub, test failures Pass ================================================================= [7128951] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:16:38 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128951/ [7128952] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload} time: 00:10:20 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128952/ [7128954] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:11:36 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128954/ [7128955] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:30:56 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128955/ [7128957] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:13:43 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128957/ [7128958] rgw/tools/{centos_latest cluster tasks} time: 02:46:06 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128958/ [7128959] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:28:36 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128959/ [7128960] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:17:42 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128960/ [7128961] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:13:27 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128961/ [7128962] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:13:29 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128962/ [7128963] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed} time: 00:09:53 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128963/ [7128964] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:14:51 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128964/ [7128966] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:18:04 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128966/ [7128967] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:34:17 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128967/ [7128968] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:40 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128968/ [7128969] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests} time: 00:13:42 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128969/ [7128970] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:14:42 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128970/ [7128971] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:12:46 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128971/ [7128972] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:41 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128972/ [7128973] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:15:27 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128973/ [7128974] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota} time: 00:09:21 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128974/ [7128976] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:15:01 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128976/ [7128977] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}} time: 00:27:06 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128977/ [7128979] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:15:02 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128979/ [7128980] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:29:25 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128980/ [7128981] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:17:05 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128981/ [7128982] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:12:58 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128982/ [7128983] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:33 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128983/ [7128984] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota} time: 00:10:21 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128984/ [7128985] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:14:36 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128985/ [7128987] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:15:59 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128987/ [7128988] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:33:22 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128988/ [7128989] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:10 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128989/ [7128990] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload} time: 00:08:59 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128990/ [7128991] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:14:11 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128991/ [7128992] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:12:56 info: http://pulpito.ceph.com/yuriw-2022-12-28_17:05:39-rgw-pacific_16.2.11_RC3.5-distro-default-smithi/7128992/