2024-03-19T13:32:36.318 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-19T13:32:36.318 INFO:teuthology.results:subset: '111/120000'
2024-03-19T13:32:36.318 INFO:teuthology.results:seed: '9945'
2024-03-19T14:08:46.976 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-19T14:08:46.999 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-19T17:59:05.234 INFO:teuthology.results:Tests finished! gathering results...
2024-03-19T17:59:08.570 INFO:scrape:Found 41 jobs
2024-03-19T17:59:11.190 INFO:scrape:Found 3 distinct failure reasons
2024-03-19T17:59:11.191 INFO:scrape:Failure: rgw multisite test failures
2024-03-19T17:59:11.191 INFO:scrape:2 jobs: ['7606577', '7606596']
2024-03-19T17:59:11.191 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2024-03-19T17:59:11.191 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2024-03-19T17:59:11.191 INFO:scrape:
2024-03-19T17:59:11.191 INFO:scrape:Failure: Command failed on smithi063 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev'
2024-03-19T17:59:11.191 INFO:scrape:1 jobs: ['7606608']
2024-03-19T17:59:11.191 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}']
2024-03-19T17:59:11.191 INFO:scrape:
2024-03-19T17:59:11.191 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi092 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test'
2024-03-19T17:59:11.192 INFO:scrape:1 jobs: ['7606597']
2024-03-19T17:59:11.192 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'supported-all-distro$/{centos_8}', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}']
2024-03-19T17:59:11.192 INFO:scrape:
2024-03-19T17:59:11.193 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  37


Fail
=================================================================
[7606577]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi}
-----------------------------------------------------------------
time:   02:01:34
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606577/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606577/
sentry: https://sentry.ceph.com/organizations/ceph/?query=506636bc87d64a22a0a25975d8d50599

    rgw multisite test failures

[7606596]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   03:53:17
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606596/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606596/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1b1c22aa7d25400489a8ce831791a4e8

    rgw multisite test failures

[7606597]  rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
-----------------------------------------------------------------
time:   00:31:37
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606597/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606597/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d1719befebb141f197b8ff88e8ee04a4

    Command failed (bucket notification tests against different endpoints) on
    smithi092 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/
    rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/s
    rc/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s
    /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v
    -a kafka_test'

[7606608]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
-----------------------------------------------------------------
time:   00:10:17
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606608/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606608/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f4b125d7e2294500ad2b4724edc6637f

    Command failed on smithi063 with status 100:
    'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install
    python-dev'



Pass
=================================================================
[7606576] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop}
time:   00:20:24
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606576/

[7606578] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:23:14
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606578/

[7606579] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:14:14
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606579/

[7606580] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:38:48
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606580/

[7606581] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:11:18
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606581/

[7606582] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:19:41
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606582/

[7606583] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:21:23
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606583/

[7606584] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:35
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606584/

[7606585] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:14:50
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606585/

[7606586] rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:17:05
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606586/

[7606587] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:08:54
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606587/

[7606588] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:19:20
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606588/

[7606589] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:12:07
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606589/

[7606590] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/cloud_transition_s3tests}
time:   00:31:42
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606590/

[7606591] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:28:29
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606591/

[7606592] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests}
time:   00:16:21
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606592/

[7606593] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop}
time:   00:22:00
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606593/

[7606594] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{rhel_8} tasks/rgw_s3tests}
time:   00:35:47
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606594/

[7606595] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:14:28
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606595/

[7606598] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest}
time:   00:16:15
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606598/

[7606599] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}
time:   00:13:04
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606599/

[7606600] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{rhel_8} tasks/{0-install first ststests}}
time:   00:37:44
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606600/

[7606601] rgw/tempest/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/rgw_tempest ubuntu_latest}
time:   00:23:50
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606601/

[7606602] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:16:59
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606602/

[7606603] rgw/tools/{centos_latest cluster ignore-pg-availability tasks}
time:   02:41:40
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606603/

[7606604] rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:16:35
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606604/

[7606605] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:40:06
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606605/

[7606606] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:23:51
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606606/

[7606607] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:12:21
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606607/

[7606609] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:15:39
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606609/

[7606610] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:13:28
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606610/

[7606611] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:12:02
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606611/

[7606612] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:11:21
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606612/

[7606613] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:18:54
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606613/

[7606614] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}}
time:   00:13:42
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606614/

[7606615] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:45
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606615/

[7606616] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:19:20
info:   https://pulpito.ceph.com/yuriw-2024-03-16_15:10:17-rgw-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606616/