2023-08-22T06:06:06.949 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-22T06:06:06.950 INFO:teuthology.results:subset: '111/12000'
2023-08-22T06:06:06.950 INFO:teuthology.results:seed: '6836'
2023-08-22T06:58:53.508 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-22T06:58:53.544 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-08-22T09:15:04.570 INFO:teuthology.results:Tests finished! gathering results...
2023-08-22T09:15:08.187 INFO:scrape:Found 43 jobs
2023-08-22T09:15:09.641 INFO:scrape:Found 2 distinct failure reasons
2023-08-22T09:15:09.642 INFO:scrape:Failure: Command failed on smithi157 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest'
2023-08-22T09:15:09.642 INFO:scrape:2 jobs: ['7375332', '7375288']
2023-08-22T09:15:09.642 INFO:scrape:suites intersection: ['overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-08-22T09:15:09.643 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-08-22T09:15:09.643 INFO:scrape:
2023-08-22T09:15:09.643 INFO:scrape:Failure: rgw multisite, test failures
2023-08-22T09:15:09.644 INFO:scrape:3 jobs: ['7375306', '7375326', '7375282']
2023-08-22T09:15:09.644 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-08-22T09:15:09.644 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}']
2023-08-22T09:15:09.645 INFO:scrape:
2023-08-22T09:15:09.647 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  38


Fail
=================================================================
[7375282]  rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi}
-----------------------------------------------------------------
time:   00:15:50
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375282/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375282/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c50baabd129490b82fc35c281fb3cc8

    rgw multisite, test failures

[7375288]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:23:29
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375288/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375288/
sentry: https://sentry.ceph.com/organizations/ceph/?query=aea60e4a1f9c48519784601800edaf92

    Command failed on smithi163 with status 1: 'cd
    /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e
    venv --notest'

[7375306]  rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}
-----------------------------------------------------------------
time:   00:51:18
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375306/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375306/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1bdcf69257a0478a9369e1144f77a231

    rgw multisite, test failures

[7375326]  rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   00:17:27
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375326/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375326/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1e4fb3aad99547f99af1ded84a763804

    rgw multisite, test failures

[7375332]  rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:23:31
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375332/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375332/
sentry: https://sentry.ceph.com/organizations/ceph/?query=98f91692b5be4ffea6d5340ea2798617

    Command failed on smithi157 with status 1: 'cd
    /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e
    venv --notest'



Pass
=================================================================
[7375276] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}
time:   00:22:31
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375276/

[7375278] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:18:11
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375278/

[7375280] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest}
time:   00:10:52
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375280/

[7375284] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}}
time:   00:16:28
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375284/

[7375286] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{rhel_8} tasks/{0-install first ststests}}
time:   00:33:01
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375286/

[7375290] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:14:41
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375290/

[7375292] rgw/tools/{centos_latest cluster tasks}
time:   02:46:36
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375292/

[7375294] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:30:44
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375294/

[7375296] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest}
time:   00:19:01
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375296/

[7375298] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:13:43
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375298/

[7375300] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}
time:   00:14:22
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375300/

[7375302] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest}
time:   00:10:28
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375302/

[7375304] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}}
time:   00:16:28
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375304/

[7375308] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:17:55
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375308/

[7375310] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:37:24
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375310/

[7375312] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}
time:   00:13:29
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375312/

[7375314] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest}
time:   00:14:40
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375314/

[7375316] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:17:25
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375316/

[7375318] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:12:52
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375318/

[7375320] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}
time:   00:13:50
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375320/

[7375322] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:16:59
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375322/

[7375324] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest}
time:   00:10:28
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375324/

[7375328] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}}
time:   00:15:10
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375328/

[7375330] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{rhel_8} tasks/{0-install first ststests}}
time:   00:33:24
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375330/

[7375334] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:14:39
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375334/

[7375336] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:30:59
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375336/

[7375338] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest}
time:   00:18:02
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375338/

[7375340] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:14:32
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375340/

[7375342] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}
time:   00:13:32
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375342/

[7375344] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest}
time:   00:11:37
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375344/

[7375346] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}
time:   00:12:03
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375346/

[7375348] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}
time:   00:41:30
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375348/

[7375350] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:17:56
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375350/

[7375352] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:35:21
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375352/

[7375354] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}
time:   00:14:06
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375354/

[7375356] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest}
time:   00:10:29
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375356/

[7375358] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}}
time:   00:15:04
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375358/

[7375360] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:14:18
info:   http://pulpito.ceph.com/yuriw-2023-08-21_23:13:17-rgw-pacific-release-distro-default-smithi/7375360/