2023-06-24T15:27:52.755 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb
2023-06-24T15:27:52.756 INFO:teuthology.results:subset: '111/12000'
2023-06-24T15:27:52.757 INFO:teuthology.results:seed: '1155'
2023-06-24T16:07:25.257 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb
2023-06-24T16:07:25.260 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-06-24T18:53:38.806 INFO:teuthology.results:Tests finished! gathering results...
2023-06-24T18:53:42.137 INFO:scrape:Found 41 jobs
2023-06-24T18:53:44.876 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315405/teuthology.log matches
2023-06-24T18:53:44.877 WARNING:scrape:Misunderstood line: 
2023-06-24T18:53:44.878 INFO:scrape:Found 4 distinct failure reasons
2023-06-24T18:53:44.879 INFO:scrape:Failure: rgw multisite test failures
2023-06-24T18:53:44.879 INFO:scrape:2 jobs: ['7315390', '7315371']
2023-06-24T18:53:44.880 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-06-24T18:53:44.881 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/test_multi}']
2023-06-24T18:53:44.881 INFO:scrape:
2023-06-24T18:53:44.882 INFO:scrape:Failure: Command failed on smithi195 with status 1: "cd /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate && openstack project create --os-username admin --os-password ADMIN --os-user-domain-id default --os-project-name admin --os-project-domain-id default --os-identity-api-version 3 --os-auth-url http://smithi195.front.sepia.ceph.com:35357/v3 --description 'Encryption Tenant' --domain default rgwcrypt --debug"
2023-06-24T18:53:44.882 INFO:scrape:1 jobs: ['7315385']
2023-06-24T18:53:44.883 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'supported-random-distro$/{rhel_8}}']
2023-06-24T18:53:44.884 INFO:scrape:
2023-06-24T18:53:44.884 INFO:scrape:Failure: Command failed on smithi131 with status 1: 'cd /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-venv/bin/activate && tox -e venv --notest'
2023-06-24T18:53:44.885 INFO:scrape:1 jobs: ['7315395']
2023-06-24T18:53:44.885 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-06-24T18:53:44.885 INFO:scrape:
2023-06-24T18:53:44.886 INFO:scrape:Valgrind: 
2023-06-24T18:53:44.886 INFO:scrape:1 jobs: ['7315405']
2023-06-24T18:53:44.886 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}']
2023-06-24T18:53:44.887 INFO:scrape:
2023-06-24T18:53:44.889 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  36


Fail
=================================================================
[7315371]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   02:11:57
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315371/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315371/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4dd51e55bd2b4b27a911de5604ca5eac

    rgw multisite test failures

[7315385]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:19:11
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315385/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315385/
sentry: https://sentry.ceph.com/organizations/ceph/?query=373ab159261d468d938a2f6126c90389

    Command failed on smithi195 with status 1: "cd
    /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate &&
    openstack project create --os-username admin --os-password ADMIN --os-user-
    domain-id default --os-project-name admin --os-project-domain-id default
    --os-identity-api-version 3 --os-auth-url
    http://smithi195.front.sepia.ceph.com:35357/v3 --description 'Encryption
    Tenant' --domain default rgwcrypt --debug"

[7315390]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_20.04} tasks/test_multi}
-----------------------------------------------------------------
time:   02:46:58
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315390/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315390/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a6e61538531740949e7f923433ca95e0

    rgw multisite test failures

[7315395]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:15:31
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315395/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315395/
sentry: https://sentry.ceph.com/organizations/ceph/?query=19df447da5cc4415896931edf9a78d8b

    Command failed on smithi131 with status 1: 'cd
    /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-
    venv/bin/activate && tox -e venv --notest'

[7315405]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   01:15:34
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315405/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315405/

    saw valgrind issues



Pass
=================================================================
[7315370] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:16:46
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315370/

[7315372] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:23:07
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315372/

[7315373] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:17
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315373/

[7315374] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:34:50
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315374/

[7315375] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:11:57
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315375/

[7315376] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:16:36
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315376/

[7315377] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:16:48
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315377/

[7315378] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:33
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315378/

[7315379] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:16:37
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315379/

[7315380] rgw/upgrade/{1-install/pacific/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:19:50
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315380/

[7315381] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:49:28
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315381/

[7315382] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:19:12
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315382/

[7315383] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:13:02
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315383/

[7315384] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/cloud_transition_s3tests}
time:   00:35:49
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315384/

[7315386] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/rgw_s3tests}
time:   00:15:59
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315386/

[7315387] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:18:25
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315387/

[7315388] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
time:   00:26:51
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315388/

[7315389] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:13:51
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315389/

[7315391] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:36:31
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315391/

[7315392] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:18:02
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315392/

[7315393] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_20.04}}
time:   00:11:55
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315393/

[7315394] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install first ststests}}
time:   00:29:12
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315394/

[7315396] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:17:27
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315396/

[7315397] rgw/tools/{centos_latest cluster tasks}
time:   02:48:50
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315397/

[7315398] rgw/upgrade/{1-install/pacific/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:18:13
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315398/

[7315399] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:34:09
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315399/

[7315400] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:23:51
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315400/

[7315401] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:12:50
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315401/

[7315402] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:21:52
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315402/

[7315403] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:18:07
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315403/

[7315404] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:30
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315404/

[7315406] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:11:51
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315406/

[7315407] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
time:   00:14:27
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315407/

[7315408] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}}
time:   00:21:36
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315408/

[7315409] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:15
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315409/

[7315410] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:16:39
info:   http://pulpito.ceph.com/yuriw-2023-06-23_21:05:04-rgw-reef-distro-default-smithi/7315410/