2023-07-15T12:25:28.698 INFO:root:teuthology version: 0.0.1.dev147+gff9991b
2023-07-15T12:25:28.699 INFO:teuthology.results:subset: '111/12000'
2023-07-15T12:25:28.700 INFO:teuthology.results:seed: '2818'
2023-07-15T13:48:07.617 INFO:root:teuthology version: 0.0.1.dev147+gff9991b
2023-07-15T13:48:07.649 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-15T16:36:21.621 INFO:teuthology.results:Tests finished! gathering results...
2023-07-15T16:36:24.962 INFO:scrape:Found 41 jobs
2023-07-15T16:36:25.311 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338875/teuthology.log matches
2023-07-15T16:36:25.312 WARNING:scrape:Misunderstood line: 
2023-07-15T16:36:26.704 INFO:scrape:Found 1 distinct failure reasons
2023-07-15T16:36:26.705 INFO:scrape:Valgrind: 
2023-07-15T16:36:26.705 INFO:scrape:4 jobs: ['7338875', '7338902', '7338845', '7338887']
2023-07-15T16:36:26.706 INFO:scrape:suites intersection: ['ignore-pg-availability']
2023-07-15T16:36:26.706 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'msgr-failures/few', 'notify', 'objectstore/bluestore-bitmap', 'omap_limits', 'overrides', 'proto/http', 'ragweed', 'realms/three-zones', 'realms/two-zonegroup', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/multisite/{clusters', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}', 'tasks/{cls', 'validater/valgrind}']
2023-07-15T16:36:26.706 INFO:scrape:
2023-07-15T16:36:26.709 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  37


Fail
=================================================================
[7338845]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}
-----------------------------------------------------------------
time:   02:20:49
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338845/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338845/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ea775470530a437587228842bc7652ee

    rgw multisite test failures

[7338875]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   01:12:48
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338875/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338875/

    saw valgrind issues

[7338887]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:19:46
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338887/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338887/
sentry: https://sentry.ceph.com/organizations/ceph/?query=465d028ac95842db9687f15d7fc94de4

    'Namespace' object has no attribute 'admin_endpoints'

[7338902]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_20.04} tasks/test_multi}
-----------------------------------------------------------------
time:   02:41:26
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338902/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338902/
sentry: https://sentry.ceph.com/organizations/ceph/?query=745a52a322ba4cbe966ed32fcb3b096b

    rgw multisite test failures



Pass
=================================================================
[7338842] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:20:54
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338842/

[7338848] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:23:26
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338848/

[7338851] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:25:17
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338851/

[7338854] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:43:10
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338854/

[7338857] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:12:10
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338857/

[7338860] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:19:11
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338860/

[7338863] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}}
time:   00:15:37
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338863/

[7338866] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:21:10
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338866/

[7338869] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:16:44
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338869/

[7338872] rgw/upgrade/{1-install/pacific/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:24:27
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338872/

[7338878] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:19:59
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338878/

[7338881] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:14:28
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338881/

[7338884] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/cloud_transition_s3tests}
time:   00:35:17
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338884/

[7338890] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:19:48
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338890/

[7338893] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:19:35
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338893/

[7338896] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:27:48
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338896/

[7338899] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:13:43
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338899/

[7338904] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:37:23
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338904/

[7338906] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:18:14
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338906/

[7338908] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_20.04}}
time:   00:12:16
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338908/

[7338910] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}}
time:   00:31:20
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338910/

[7338912] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
time:   00:26:32
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338912/

[7338914] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:19:56
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338914/

[7338916] rgw/tools/{centos_latest cluster tasks}
time:   02:53:59
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338916/

[7338918] rgw/upgrade/{1-install/pacific/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:21:35
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338918/

[7338920] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:41:26
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338920/

[7338922] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:24:07
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338922/

[7338924] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:13:34
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338924/

[7338926] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:20:41
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338926/

[7338928] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:17:32
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338928/

[7338930] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:24:21
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338930/

[7338932] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:01:23
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338932/

[7338934] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:12:14
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338934/

[7338936] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:25:24
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338936/

[7338938] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_20.04}}
time:   00:12:15
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338938/

[7338940] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:22:44
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338940/

[7338942] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:19:55
info:   http://pulpito.ceph.com/yuriw-2023-07-14_23:46:07-rgw-reef-distro-default-smithi/7338942/