2023-07-23T20:40:30.807 INFO:root:teuthology version: 0.0.1.dev158+g1426592
2023-07-23T20:40:30.808 INFO:teuthology.results:subset: '111/12000'
2023-07-23T20:40:30.808 INFO:teuthology.results:seed: '9063'
2023-07-23T22:34:42.288 INFO:root:teuthology version: 0.0.1.dev158+g1426592
2023-07-23T22:34:42.316 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-24T01:14:00.496 INFO:teuthology.results:Tests finished! gathering results...
2023-07-24T01:14:04.132 INFO:scrape:Found 41 jobs
2023-07-24T01:14:07.351 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348714/teuthology.log matches
2023-07-24T01:14:07.352 WARNING:scrape:Misunderstood line: 
2023-07-24T01:14:07.712 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348762/teuthology.log matches
2023-07-24T01:14:07.713 WARNING:scrape:Misunderstood line: 
2023-07-24T01:14:07.713 INFO:scrape:Found 3 distinct failure reasons
2023-07-24T01:14:07.713 INFO:scrape:Failure: rgw multisite test failures
2023-07-24T01:14:07.714 INFO:scrape:2 jobs: ['7348732', '7348692']
2023-07-24T01:14:07.714 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/test_multi}']
2023-07-24T01:14:07.714 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/test_multi}']
2023-07-24T01:14:07.715 INFO:scrape:
2023-07-24T01:14:07.715 INFO:scrape:Failure: 'Namespace' object has no attribute 'admin_endpoints'
2023-07-24T01:14:07.715 INFO:scrape:1 jobs: ['7348722']
2023-07-24T01:14:07.715 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'supported-random-distro$/{rhel_8}}']
2023-07-24T01:14:07.716 INFO:scrape:
2023-07-24T01:14:07.717 INFO:scrape:Valgrind: 
2023-07-24T01:14:07.717 INFO:scrape:2 jobs: ['7348714', '7348762']
2023-07-24T01:14:07.718 INFO:scrape:suites intersection: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-07-24T01:14:07.718 INFO:scrape:suites union: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-07-24T01:14:07.718 INFO:scrape:
2023-07-24T01:14:07.721 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  36


Fail
=================================================================
[7348692]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi}
-----------------------------------------------------------------
time:   03:21:00
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348692/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348692/
sentry: https://sentry.ceph.com/organizations/ceph/?query=741b5d542ae844ef9230d991b3112d14

    rgw multisite test failures

[7348714]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
-----------------------------------------------------------------
time:   01:19:09
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348714/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348714/

    saw valgrind issues

[7348722]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:29:20
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348722/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348722/
sentry: https://sentry.ceph.com/organizations/ceph/?query=60d7958e942e4badbee427ea2764c85c

    'Namespace' object has no attribute 'admin_endpoints'

[7348732]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_20.04} tasks/test_multi}
-----------------------------------------------------------------
time:   02:27:13
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348732/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348732/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8021ab72e168401aae585e82fcef3551

    rgw multisite test failures

[7348762]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
-----------------------------------------------------------------
time:   01:04:17
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348762/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348762/

    saw valgrind issues



Pass
=================================================================
[7348689] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:18:27
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348689/

[7348695] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:24:09
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348695/

[7348698] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:22:39
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348698/

[7348700] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:37:31
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348700/

[7348702] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:13:38
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348702/

[7348704] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
time:   00:16:37
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348704/

[7348706] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:24:04
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348706/

[7348708] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:23:14
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348708/

[7348710] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:18:41
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348710/

[7348712] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:15:39
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348712/

[7348716] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:21:17
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348716/

[7348718] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:14:58
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348718/

[7348720] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/cloud_transition_s3tests}
time:   00:34:35
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348720/

[7348724] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests}
time:   00:22:55
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348724/

[7348726] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:18:58
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348726/

[7348728] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:27:58
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348728/

[7348730] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:14:08
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348730/

[7348734] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:36:21
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348734/

[7348736] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:17:03
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348736/

[7348738] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}}
time:   00:15:26
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348738/

[7348740] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install first ststests}}
time:   00:29:26
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348740/

[7348742] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
time:   00:25:43
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348742/

[7348744] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:17:29
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348744/

[7348746] rgw/tools/{centos_latest cluster tasks}
time:   02:46:45
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348746/

[7348748] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:14:22
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348748/

[7348750] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:36:33
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348750/

[7348752] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:24:03
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348752/

[7348754] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:13:05
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348754/

[7348756] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:20:01
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348756/

[7348758] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:16:22
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348758/

[7348760] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:39
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348760/

[7348764] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:12:16
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348764/

[7348766] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:19:05
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348766/

[7348768] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_20.04}}
time:   00:12:17
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348768/

[7348770] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:46
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348770/

[7348772] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:19:06
info:   http://pulpito.ceph.com/yuriw-2023-07-23_14:04:13-rgw-reef-distro-default-smithi/7348772/