2023-11-06T06:17:56.057 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-06T06:17:56.058 INFO:teuthology.results:subset: '111/12000'
2023-11-06T06:17:56.058 INFO:teuthology.results:seed: '3989'
2023-11-06T07:03:16.944 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-06T07:03:16.947 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-06T09:42:29.570 INFO:teuthology.results:Tests finished! gathering results...
2023-11-06T09:42:32.617 INFO:scrape:Found 41 jobs
2023-11-06T09:42:35.628 INFO:scrape:Found 2 distinct failure reasons
2023-11-06T09:42:35.629 INFO:scrape:Failure: rgw multisite test failures
2023-11-06T09:42:35.629 INFO:scrape:2 jobs: ['7449031', '7449012']
2023-11-06T09:42:35.629 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}']
2023-11-06T09:42:35.630 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}']
2023-11-06T09:42:35.630 INFO:scrape:
2023-11-06T09:42:35.631 INFO:scrape:Failure: Command failed on smithi082 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev'
2023-11-06T09:42:35.631 INFO:scrape:1 jobs: ['7449043']
2023-11-06T09:42:35.631 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}']
2023-11-06T09:42:35.632 INFO:scrape:
2023-11-06T09:42:35.634 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/
failed:  3
dead:    0
running: 0
waiting: 0
queued:  0
passed:  38


Fail
=================================================================
[7449012]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   02:25:33
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449012/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449012/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3ae97626a54040a4a4dd05a4c364c431

    rgw multisite test failures

[7449031]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   02:30:32
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449031/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449031/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d8bb4de6e40648ef82f274ca85e4129e

    rgw multisite test failures

[7449043]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
-----------------------------------------------------------------
time:   00:10:40
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449043/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449043/
sentry: https://sentry.ceph.com/organizations/ceph/?query=45d560078d5241b59893ead92d447213

    Command failed on smithi082 with status 100:
    'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install
    python-dev'



Pass
=================================================================
[7449011] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop}
time:   00:17:18
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449011/

[7449013] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:25:51
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449013/

[7449014] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:14:32
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449014/

[7449015] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:39:27
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449015/

[7449016] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:12:18
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449016/

[7449017] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:20:04
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449017/

[7449018] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:17:14
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449018/

[7449019] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:31
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449019/

[7449020] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:16:11
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449020/

[7449021] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:13:45
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449021/

[7449022] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_8} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:57:11
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449022/

[7449023] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:20:17
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449023/

[7449024] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:13:11
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449024/

[7449025] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests}
time:   00:36:00
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449025/

[7449026] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:28:59
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449026/

[7449027] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests}
time:   00:16:42
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449027/

[7449028] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop}
time:   00:18:26
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449028/

[7449029] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/rgw_s3tests}
time:   00:25:28
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449029/

[7449030] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:13:54
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449030/

[7449032] rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:36:28
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449032/

[7449033] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest}
time:   00:17:25
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449033/

[7449034] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_latest}}
time:   00:12:11
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449034/

[7449035] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install first ststests}}
time:   00:30:06
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449035/

[7449036] rgw/tempest/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/rgw_tempest ubuntu_latest}
time:   00:25:49
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449036/

[7449037] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:16:59
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449037/

[7449038] rgw/tools/{centos_latest cluster ignore-pg-availability tasks}
time:   02:43:23
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449038/

[7449039] rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:19:01
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449039/

[7449040] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:38:43
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449040/

[7449041] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:23:44
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449041/

[7449042] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:13:08
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449042/

[7449044] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:16:48
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449044/

[7449045] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:13:58
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449045/

[7449046] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:20:03
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449046/

[7449047] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:11:53
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449047/

[7449048] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:19:59
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449048/

[7449049] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_20.04}}
time:   00:12:17
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449049/

[7449050] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:21:11
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449050/

[7449051] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:19:50
info:   http://pulpito.ceph.com/yuriw-2023-11-05_23:51:23-rgw-reef-release-distro-default-smithi/7449051/