2023-09-30T07:09:27.054 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-30T07:09:27.054 INFO:teuthology.results:subset: '111/12000'
2023-09-30T07:09:27.054 INFO:teuthology.results:seed: '3870'
2023-09-30T08:13:07.898 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-30T08:13:07.929 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-30T19:55:03.405 INFO:teuthology.results:Tests finished! gathering results...
2023-09-30T19:55:06.698 INFO:scrape:Found 42 jobs
2023-09-30T19:55:07.929 INFO:scrape:Found 4 distinct failure reasons
2023-09-30T19:55:07.930 INFO:scrape:Failure: rgw multisite test failures
2023-09-30T19:55:07.930 INFO:scrape:2 jobs: ['7407010', '7407070']
2023-09-30T19:55:07.930 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-09-30T19:55:07.930 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-09-30T19:55:07.930 INFO:scrape:
2023-09-30T19:55:07.930 INFO:scrape:Failure: Command failed (ragweed tests against rgw) on smithi157 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/ragweed.client.0 && RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg tox --sitepackages -- -v -m 'not fails_on_rgw'"
2023-09-30T19:55:07.930 INFO:scrape:2 jobs: ['7407016', '7407094']
2023-09-30T19:55:07.931 INFO:scrape:suites intersection: ['2-setup', '3-upgrade-sequence/osds-then-rgws', 'cluster', 'frontend/beast', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'overrides}']
2023-09-30T19:55:07.931 INFO:scrape:suites union: ['2-setup', '3-upgrade-sequence/osds-then-rgws', 'cluster', 'frontend/beast', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'overrides}', 'rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04}', 'rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04}']
2023-09-30T19:55:07.931 INFO:scrape:
2023-09-30T19:55:07.931 INFO:scrape:Dead: b'2023-09-30T19:51:21.135 DEBUG:teuthology.exit:Finished running handlers'
2023-09-30T19:55:07.931 INFO:scrape:1 jobs: ['7407073']
2023-09-30T19:55:07.931 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'supported-all-distro$/{centos_8}', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}']
2023-09-30T19:55:07.931 INFO:scrape:
2023-09-30T19:55:07.932 INFO:scrape:Failure: Command failed on smithi170 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev'
2023-09-30T19:55:07.932 INFO:scrape:1 jobs: ['7407106']
2023-09-30T19:55:07.932 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}']
2023-09-30T19:55:07.932 INFO:scrape:
2023-09-30T19:55:07.934 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/
failed:  5
dead:    1
running: 0
waiting: 0
queued:  0
passed:  36


Fail
=================================================================
[7407010]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:17:57
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407010/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407010/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c06f2ce9b644565b80464a923458d21

    rgw multisite test failures

[7407016]  rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
-----------------------------------------------------------------
time:   00:16:05
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407016/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407016/
sentry: https://sentry.ceph.com/organizations/ceph/?query=768b45fb2ee442faa014f5d015f323e0

    Command failed (ragweed tests against rgw) on smithi157 with status 1:
    "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/ragweed.client.0 &&
    RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf
    RAGWEED_STAGES=check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg tox
    --sitepackages -- -v -m 'not fails_on_rgw'"

[7407070]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   04:48:41
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407070/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407070/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bc3fa662e0744d3b9d8f26118e4ed2ba

    rgw multisite test failures

[7407094]  rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
-----------------------------------------------------------------
time:   00:16:22
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407094/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407094/
sentry: https://sentry.ceph.com/organizations/ceph/?query=103da644fed4401e9e2d7069b4fe14be

    Command failed (ragweed tests against rgw) on smithi049 with status 1:
    "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd
    /home/ubuntu/cephtest/ragweed.client.0 &&
    RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf
    RAGWEED_STAGES=check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg tox
    --sitepackages -- -v -m 'not fails_on_rgw'"

[7407106]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
-----------------------------------------------------------------
time:   00:11:00
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407106/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407106/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ffe151d925574c8b8bef6c52cbdbd7e3

    Command failed on smithi170 with status 100:
    'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install
    python-dev'



Dead
=================================================================
[7407073]  rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407073/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407073/

    hit max job timeout



Pass
=================================================================
[7407013] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:22:31
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407013/

[7407019] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:38:43
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407019/

[7407022] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:17:32
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407022/

[7407025] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:25:30
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407025/

[7407028] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_latest}}
time:   00:15:49
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407028/

[7407031] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:24:25
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407031/

[7407034] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:16:13
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407034/

[7407037] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:16:43
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407037/

[7407040] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_8} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:56:23
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407040/

[7407043] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:25:09
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407043/

[7407046] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:14:34
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407046/

[7407049] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/cloud_transition_s3tests}
time:   00:32:47
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407049/

[7407052] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:35:10
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407052/

[7407056] rgw/d4n/{cluster ignore-pg-availability overrides supported-random-distro$/{ubuntu_20.04} tasks/rgw_d4ntests}
time:   00:15:12
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407056/

[7407058] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests}
time:   00:22:26
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407058/

[7407061] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop}
time:   00:17:43
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407061/

[7407064] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/rgw_s3tests}
time:   00:37:53
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407064/

[7407067] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:16:28
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407067/

[7407076] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest}
time:   00:17:00
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407076/

[7407079] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}}
time:   00:20:37
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407079/

[7407082] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_latest} tasks/{0-install 1-keycloak 2-s3tests}}
time:   00:36:00
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407082/

[7407085] rgw/tempest/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/rgw_tempest ubuntu_latest}
time:   00:25:41
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407085/

[7407088] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:17:18
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407088/

[7407091] rgw/tools/{centos_latest cluster ignore-pg-availability tasks}
time:   02:40:40
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407091/

[7407097] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:39:32
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407097/

[7407100] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:23:15
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407100/

[7407103] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:15:15
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407103/

[7407109] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:16:49
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407109/

[7407112] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:17:02
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407112/

[7407115] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:20:14
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407115/

[7407118] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:18:10
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407118/

[7407121] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:26:08
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407121/

[7407124] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}}
time:   00:19:34
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407124/

[7407127] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:23:59
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407127/

[7407130] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:25:22
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407130/

[7407133] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop}
time:   00:18:48
info:   http://pulpito.ceph.com/yuriw-2023-09-29_19:43:20-rgw-main-distro-default-smithi/7407133/