2023-07-01T03:19:18.854 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-01T03:19:18.855 INFO:teuthology.results:subset: '111/12000'
2023-07-01T03:19:18.855 INFO:teuthology.results:seed: '6738'
2023-07-01T04:13:53.573 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-01T04:13:53.596 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-01T06:56:07.100 INFO:teuthology.results:Tests finished! gathering results...
2023-07-01T06:56:10.452 INFO:scrape:Found 41 jobs
2023-07-01T06:56:13.222 INFO:scrape:Found 2 distinct failure reasons
2023-07-01T06:56:13.223 INFO:scrape:Failure: Command failed on smithi057 with status 1: 'cd /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-venv/bin/activate && tox -e venv --notest'
2023-07-01T06:56:13.224 INFO:scrape:1 jobs: ['7323618']
2023-07-01T06:56:13.224 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-07-01T06:56:13.224 INFO:scrape:
2023-07-01T06:56:13.225 INFO:scrape:Failure: rgw multisite test failures
2023-07-01T06:56:13.225 INFO:scrape:2 jobs: ['7323608', '7323555']
2023-07-01T06:56:13.225 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-07-01T06:56:13.226 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-07-01T06:56:13.226 INFO:scrape:
2023-07-01T06:56:13.229 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/
failed:  3
dead:    0
running: 0
waiting: 0
queued:  0
passed:  38


Fail
=================================================================
[7323555]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:16:09
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323555/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323555/
sentry: https://sentry.ceph.com/organizations/ceph/?query=656449f21249474388e0076baa7e6ae0

    rgw multisite test failures

[7323608]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:51:51
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323608/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323608/
sentry: https://sentry.ceph.com/organizations/ceph/?query=70bf18060100499fb5a821fcce111ca2

    rgw multisite test failures

[7323618]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:15:52
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323618/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323618/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ae0321fcda8c482f88989410b68561d9

    Command failed on smithi057 with status 1: 'cd
    /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-
    venv/bin/activate && tox -e venv --notest'



Pass
=================================================================
[7323552] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:16:40
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323552/

[7323558] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:23:17
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323558/

[7323561] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:14:08
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323561/

[7323563] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:43:04
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323563/

[7323566] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:11:50
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323566/

[7323569] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:19:22
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323569/

[7323572] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:21:21
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323572/

[7323575] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:01
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323575/

[7323578] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:16:21
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323578/

[7323581] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:41
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323581/

[7323584] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:48:07
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323584/

[7323587] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
time:   00:15:19
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323587/

[7323590] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:13:12
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323590/

[7323593] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/cloud_transition_s3tests}
time:   00:39:50
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323593/

[7323596] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
time:   00:23:06
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323596/

[7323599] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests}
time:   00:21:10
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323599/

[7323602] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:18:50
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323602/

[7323604] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests}
time:   00:30:25
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323604/

[7323606] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:14:06
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323606/

[7323610] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:37:18
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323610/

[7323612] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:17:20
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323612/

[7323614] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_20.04}}
time:   00:12:00
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323614/

[7323616] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}}
time:   00:31:09
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323616/

[7323620] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:17:03
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323620/

[7323622] rgw/tools/{centos_latest cluster tasks}
time:   02:47:44
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323622/

[7323624] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:14:01
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323624/

[7323626] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:41:44
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323626/

[7323628] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:22:40
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323628/

[7323630] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:13:10
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323630/

[7323632] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:17:43
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323632/

[7323634] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:16:44
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323634/

[7323636] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:14:31
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323636/

[7323638] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   01:12:58
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323638/

[7323640] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:12:01
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323640/

[7323642] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:18:37
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323642/

[7323644] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}
time:   00:13:48
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323644/

[7323646] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:17
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323646/

[7323648] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
time:   00:15:16
info:   http://pulpito.ceph.com/yuriw-2023-06-30_22:02:53-rgw-reef-distro-default-smithi/7323648/