2023-12-15T13:52:01.337 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-15T13:52:01.337 INFO:teuthology.results:subset: '111/12000' 2023-12-15T13:52:01.338 INFO:teuthology.results:seed: '8526' 2023-12-15T14:42:06.440 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-15T14:42:06.461 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-12-15T16:55:17.387 INFO:teuthology.results:Tests finished! gathering results... 2023-12-15T16:55:20.723 INFO:scrape:Found 43 jobs 2023-12-15T16:55:22.174 INFO:scrape:Found 2 distinct failure reasons 2023-12-15T16:55:22.174 INFO:scrape:Failure: rgw multisite, test failures 2023-12-15T16:55:22.174 INFO:scrape:3 jobs: ['7492697', '7492685', '7492707'] 2023-12-15T16:55:22.174 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2023-12-15T16:55:22.175 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2023-12-15T16:55:22.175 INFO:scrape: 2023-12-15T16:55:22.175 INFO:scrape:Failure: Command failed on smithi116 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' 2023-12-15T16:55:22.175 INFO:scrape:2 jobs: ['7492710', '7492688'] 2023-12-15T16:55:22.175 INFO:scrape:suites intersection: ['overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-12-15T16:55:22.175 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-12-15T16:55:22.175 INFO:scrape: 2023-12-15T16:55:22.177 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 38 Fail ================================================================= [7492685] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:15:45 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492685/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492685/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e5cb0192d194b5297b93a3cbf446379 rgw multisite, test failures [7492688] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:22:05 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492688/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492688/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4bbf8821a89c4bbaac66d9233b90abca Command failed on smithi019 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7492697] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:41:17 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492697/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492697/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a16faff200b2469c82a60eb1e31ebef0 rgw multisite, test failures [7492707] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:16:38 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492707/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492707/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c24964cd1c54303b0d20f7e76cd4e6b rgw multisite, test failures [7492710] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:20:29 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492710/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492710/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e55a00efd524765a4dbb0c354ec3dde Command failed on smithi116 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' Pass ================================================================= [7492682] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:22:59 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492682/ [7492683] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:18:35 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492683/ [7492684] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:48 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492684/ [7492686] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:12:01 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492686/ [7492687] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:31:24 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492687/ [7492689] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:15:24 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492689/ [7492690] rgw/tools/{centos_latest cluster tasks} time: 02:46:23 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492690/ [7492691] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:31:53 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492691/ [7492692] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:19:11 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492692/ [7492693] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:14:31 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492693/ [7492694] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:15:26 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492694/ [7492695] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:09:59 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492695/ [7492696] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:11:56 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492696/ [7492698] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:19:03 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492698/ [7492699] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 00:36:35 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492699/ [7492700] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:01 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492700/ [7492701] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:15:11 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492701/ [7492702] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:14:44 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492702/ [7492703] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:49 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492703/ [7492704] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:04 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492704/ [7492705] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:17:34 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492705/ [7492706] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:10:20 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492706/ [7492708] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:14:42 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492708/ [7492709] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:31:01 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492709/ [7492712] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:37 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492712/ [7492714] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:31:16 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492714/ [7492716] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:18:30 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492716/ [7492718] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:17:32 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492718/ [7492720] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:39 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492720/ [7492722] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:11:42 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492722/ [7492724] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:15:14 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492724/ [7492726] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} time: 00:32:25 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492726/ [7492728] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:56 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492728/ [7492730] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 00:35:42 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492730/ [7492732] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:17 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492732/ [7492733] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:56 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492733/ [7492735] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:11:41 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492735/ [7492737] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:14:29 info: https://pulpito.ceph.com/yuriw-2023-12-14_22:28:41-rgw-wip-yuri5-testing-2023-12-14-1107-pacific-distro-default-smithi/7492737/