2023-07-26T14:55:00.880 INFO:root:teuthology version: 0.0.1.dev160+g407880c
2023-07-26T14:55:00.881 INFO:teuthology.results:subset: '111/12000'
2023-07-26T14:55:00.881 INFO:teuthology.results:seed: '5272'
2023-07-26T15:40:25.827 INFO:root:teuthology version: 0.0.1.dev160+g407880c
2023-07-26T15:40:25.850 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-26T18:24:39.292 INFO:teuthology.results:Tests finished! gathering results...
2023-07-26T18:24:42.302 INFO:scrape:Found 41 jobs
2023-07-26T18:24:42.475 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351840/teuthology.log
2023-07-26T18:24:44.558 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351862/teuthology.log matches
2023-07-26T18:24:44.560 WARNING:scrape:Misunderstood line: 
2023-07-26T18:24:44.675 INFO:scrape:Found 4 distinct failure reasons
2023-07-26T18:24:44.675 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi122 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'"
2023-07-26T18:24:44.676 INFO:scrape:1 jobs: ['7351859']
2023-07-26T18:24:44.676 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'supported-random-distro$/{ubuntu_20.04}}']
2023-07-26T18:24:44.676 INFO:scrape:
2023-07-26T18:24:44.677 INFO:scrape:Dead: None
2023-07-26T18:24:44.677 INFO:scrape:4 jobs: ['7351840', '7351856', '7351828', '7351853']
2023-07-26T18:24:44.677 INFO:scrape:
2023-07-26T18:24:44.678 INFO:scrape:Failure: rgw multisite test failures
2023-07-26T18:24:44.678 INFO:scrape:1 jobs: ['7351847']
2023-07-26T18:24:44.678 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'tasks/test_multi}']
2023-07-26T18:24:44.678 INFO:scrape:
2023-07-26T18:24:44.679 INFO:scrape:Valgrind: 
2023-07-26T18:24:44.679 INFO:scrape:2 jobs: ['7351862', '7351842']
2023-07-26T18:24:44.680 INFO:scrape:suites intersection: ['ignore-pg-availability', 's3tests-branch']
2023-07-26T18:24:44.680 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{centos_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']
2023-07-26T18:24:44.680 INFO:scrape:
2023-07-26T18:24:44.683 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/
failed:  4
dead:    4
running: 0
waiting: 0
queued:  0
passed:  33


Fail
=================================================================
[7351842]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:24:43
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351842/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351842/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cc05b1c5a93f4d8e88a746fb800c7290

    'Namespace' object has no attribute 'admin_endpoints'

[7351847]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:52:39
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351847/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351847/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3c4138cf3fb747b29ec60e5995eb6789

    rgw multisite test failures

[7351859]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
-----------------------------------------------------------------
time:   00:18:13
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351859/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351859/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8666d234f9e14c5ab94cf31a0adb7489

    Command failed (s3 tests against rgw) on smithi122 with status 1: "source
    /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/s3-tests-client.0 &&
    S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not
    fails_on_rgw and not lifecycle_expiration and not test_of_sts and not
    webidentity_test and not fails_with_subdomain and not sse_s3'"

[7351862]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
-----------------------------------------------------------------
time:   00:54:18
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351862/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351862/

    saw valgrind issues



Dead
=================================================================
[7351828]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351828/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351828/

    Error reimaging machines: Failed to power on smithi084

[7351840]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351840/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351840/

    Error reimaging machines: Failed to power on smithi077

[7351853]  rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351853/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351853/

    Error reimaging machines: Failed to power on smithi188

[7351856]  rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351856/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351856/

    Error reimaging machines: Expected smithi018's OS to be centos 9 but found
    ubuntu 22.04



Pass
=================================================================
[7351827] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:36:09
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351827/

[7351829] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:28:56
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351829/

[7351830] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:22:37
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351830/

[7351831] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:45:34
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351831/

[7351832] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:16:44
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351832/

[7351833] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:27:25
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351833/

[7351834] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}}
time:   00:22:16
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351834/

[7351835] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:24:53
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351835/

[7351836] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:23:30
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351836/

[7351837] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:22:48
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351837/

[7351838] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:01:25
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351838/

[7351839] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_latest}}
time:   00:18:11
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351839/

[7351841] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/cloud_transition_s3tests}
time:   00:33:44
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351841/

[7351843] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
time:   00:22:45
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351843/

[7351844] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:26:55
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351844/

[7351845] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:30:24
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351845/

[7351846] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:15:31
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351846/

[7351848] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:39:52
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351848/

[7351849] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:19:14
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351849/

[7351850] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}
time:   00:14:58
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351850/

[7351851] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{rhel_8} tasks/{0-install first ststests}}
time:   00:38:58
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351851/

[7351852] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
time:   00:26:27
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351852/

[7351854] rgw/tools/{centos_latest cluster tasks}
time:   02:52:31
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351854/

[7351855] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:15:49
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351855/

[7351857] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:23:36
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351857/

[7351858] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:14:23
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351858/

[7351860] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:17:43
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351860/

[7351861] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:15:00
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351861/

[7351863] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:12:42
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351863/

[7351864] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:24:08
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351864/

[7351865] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_20.04}}
time:   00:12:27
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351865/

[7351866] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:25
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351866/

[7351867] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
time:   00:20:10
info:   http://pulpito.ceph.com/yuriw-2023-07-25_15:15:44-rgw-reef-release-distro-default-smithi/7351867/