2023-06-08T15:10:27.427 INFO:root:teuthology version: 0.0.1.dev120+g2ef0dcd
2023-06-08T15:10:27.429 INFO:teuthology.results:subset: '111/12000'
2023-06-08T15:10:27.429 INFO:teuthology.results:seed: '3386'
2023-06-08T15:44:51.849 INFO:root:teuthology version: 0.0.1.dev120+g2ef0dcd
2023-06-08T15:44:51.885 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-06-08T18:32:05.626 INFO:teuthology.results:Tests finished! gathering results...
2023-06-08T18:32:08.632 INFO:scrape:Found 41 jobs
2023-06-08T18:32:09.408 INFO:scrape:Found 2 distinct failure reasons
2023-06-08T18:32:09.409 INFO:scrape:Failure: Command failed on smithi160 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest'
2023-06-08T18:32:09.409 INFO:scrape:1 jobs: ['7298771']
2023-06-08T18:32:09.409 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-06-08T18:32:09.410 INFO:scrape:
2023-06-08T18:32:09.410 INFO:scrape:Failure: rgw multisite test failures
2023-06-08T18:32:09.410 INFO:scrape:1 jobs: ['7298767']
2023-06-08T18:32:09.411 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}']
2023-06-08T18:32:09.411 INFO:scrape:
2023-06-08T18:32:09.413 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/
failed:  2
dead:    0
running: 0
waiting: 0
queued:  0
passed:  39


Fail
=================================================================
[7298767]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   01:04:00
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298767/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298767/
sentry: https://sentry.ceph.com/organizations/ceph/?query=44c2e4d7c3bf4295830e2a48dd48feb2

    rgw multisite test failures

[7298771]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:23:55
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298771/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298771/
sentry: https://sentry.ceph.com/organizations/ceph/?query=20f703e2400e47efb0e08da3ab07b5cf

    Command failed on smithi160 with status 1: 'cd
    /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e
    venv --notest'



Pass
=================================================================
[7298762] rgw/cloud-transition/{cluster overrides supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests}
time:   00:34:03
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298762/

[7298763] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}}
time:   00:28:16
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298763/

[7298764] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:22:15
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298764/

[7298765] rgw/lifecycle/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests}
time:   00:32:04
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298765/

[7298766] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest}
time:   00:13:31
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298766/

[7298768] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:44:03
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298768/

[7298769] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}}
time:   00:17:50
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298769/

[7298770] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8} tasks/{0-install first ststests}}
time:   00:39:53
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298770/

[7298772] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:16:27
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298772/

[7298773] rgw/tools/{centos_latest cluster tasks}
time:   02:47:08
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298773/

[7298774] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:30:59
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298774/

[7298775] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest}
time:   00:19:13
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298775/

[7298776] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:13:39
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298776/

[7298777] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}}
time:   00:18:17
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298777/

[7298778] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest}
time:   00:10:28
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298778/

[7298779] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}}
time:   00:20:32
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298779/

[7298780] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:22:45
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298780/

[7298781] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:02:02
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298781/

[7298782] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}}
time:   00:20:52
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298782/

[7298783] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest}
time:   00:15:52
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298783/

[7298784] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:19:06
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298784/

[7298785] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:14:54
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298785/

[7298786] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}}
time:   00:17:17
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298786/

[7298787] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:18:00
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298787/

[7298788] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest}
time:   00:10:59
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298788/

[7298789] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}
time:   00:35:15
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298789/

[7298790] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}
time:   00:12:33
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298790/

[7298791] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:15:48
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298791/

[7298792] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:32:07
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298792/

[7298793] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:14:51
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298793/

[7298794] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}}
time:   00:17:22
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298794/

[7298795] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest}
time:   00:13:06
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298795/

[7298796] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}}
time:   00:24:19
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298796/

[7298797] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:19:05
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298797/

[7298798] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:02:58
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298798/

[7298799] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}}
time:   00:15:09
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298799/

[7298800] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest}
time:   00:11:43
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298800/

[7298801] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:16:46
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298801/

[7298802] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:16:55
info:   http://pulpito.ceph.com/yuriw-2023-06-08_15:10:10-rgw-wip-yuri8-testing-2023-06-07-1323-quincy-distro-default-smithi/7298802/