2023-07-24T14:47:00.550 INFO:root:teuthology version: 0.0.1.dev158+g1426592
2023-07-24T14:47:00.551 INFO:teuthology.results:subset: '111/120000'
2023-07-24T14:47:00.551 INFO:teuthology.results:seed: '9663'
2023-07-24T15:18:53.428 INFO:root:teuthology version: 0.0.1.dev158+g1426592
2023-07-24T15:18:53.432 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-24T18:57:09.713 INFO:teuthology.results:Tests finished! gathering results...
2023-07-24T18:57:12.925 INFO:scrape:Found 41 jobs
2023-07-24T18:57:14.486 INFO:scrape:Found 2 distinct failure reasons
2023-07-24T18:57:14.487 INFO:scrape:Failure: rgw multisite test failures
2023-07-24T18:57:14.488 INFO:scrape:2 jobs: ['7349776', '7349795']
2023-07-24T18:57:14.488 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'tasks/test_multi}']
2023-07-24T18:57:14.489 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'tasks/test_multi}']
2023-07-24T18:57:14.489 INFO:scrape:
2023-07-24T18:57:14.489 INFO:scrape:Failure: 'Namespace' object has no attribute 'admin_endpoints'
2023-07-24T18:57:14.490 INFO:scrape:1 jobs: ['7349790']
2023-07-24T18:57:14.490 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'supported-random-distro$/{rhel_8}}']
2023-07-24T18:57:14.490 INFO:scrape:
2023-07-24T18:57:14.493 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/
failed:  3
dead:    0
running: 0
waiting: 0
queued:  0
passed:  38


Fail
=================================================================
[7349776]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}
-----------------------------------------------------------------
time:   02:13:16
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349776/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349776/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dda7988071714679982023a286e3be09

    rgw multisite test failures

[7349790]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:26:21
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349790/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349790/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ee1f8d4538fc4cadaacae38d848dea13

    'Namespace' object has no attribute 'admin_endpoints'

[7349795]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi}
-----------------------------------------------------------------
time:   03:43:55
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349795/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349795/
sentry: https://sentry.ceph.com/organizations/ceph/?query=abe4d93db73b4d2ea6eb47bbfd509204

    rgw multisite test failures



Pass
=================================================================
[7349775] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:35:35
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349775/

[7349777] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:26:54
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349777/

[7349778] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:24:36
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349778/

[7349779] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:45:43
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349779/

[7349780] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:14:48
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349780/

[7349781] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:24:00
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349781/

[7349782] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_20.04}}
time:   00:14:58
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349782/

[7349783] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:25:46
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349783/

[7349784] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:23:30
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349784/

[7349785] rgw/upgrade/{1-install/pacific/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:38:40
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349785/

[7349786] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:24:24
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349786/

[7349787] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:24:04
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349787/

[7349788] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:18:16
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349788/

[7349789] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/cloud_transition_s3tests}
time:   00:34:58
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349789/

[7349791] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
time:   00:26:36
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349791/

[7349792] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:33:40
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349792/

[7349793] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
time:   00:32:54
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349793/

[7349794] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:21:06
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349794/

[7349796] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:55:46
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349796/

[7349797] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:22:13
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349797/

[7349798] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}}
time:   00:17:35
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349798/

[7349799] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}}
time:   00:34:56
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349799/

[7349800] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
time:   00:29:27
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349800/

[7349801] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:21:17
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349801/

[7349802] rgw/tools/{centos_latest cluster tasks}
time:   02:52:52
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349802/

[7349803] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:19:33
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349803/

[7349804] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:43:00
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349804/

[7349805] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:26:24
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349805/

[7349806] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:16:54
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349806/

[7349807] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:20:04
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349807/

[7349808] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:18:19
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349808/

[7349809] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:19:59
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349809/

[7349810] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:53:17
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349810/

[7349811] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:13:14
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349811/

[7349812] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:23:39
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349812/

[7349813] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}
time:   00:14:57
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349813/

[7349814] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:21:09
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349814/

[7349815] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
time:   00:16:17
info:   http://pulpito.ceph.com/yuriw-2023-07-24_14:37:51-rgw-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7349815/