2023-04-01T00:36:46.684 INFO:root:teuthology version: 0.0.1.dev91+g5338cb9
2023-04-01T00:36:46.685 INFO:teuthology.results:subset: '111/12000'
2023-04-01T00:36:46.685 INFO:teuthology.results:seed: '6429'
2023-04-01T01:19:32.390 INFO:root:teuthology version: 0.0.1.dev91+g5338cb9
2023-04-01T01:19:32.408 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-01T11:31:23.087 INFO:teuthology.results:Tests finished! gathering results...
2023-04-01T11:31:26.001 INFO:scrape:Found 37 jobs
2023-04-01T11:31:27.007 WARNING:scrape:Not-understood crash indication 2023-04-01T02:41:16.155 INFO:tasks.rgw.c1.client.0.smithi118.stderr:daemon-helper: command crashed with signal 6
2023-04-01T11:31:27.008 WARNING:scrape:Not-understood crash indication 2023-04-01T10:29:27.057 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi118.stderr: INFO: daemon-helper: command crashed with signal 6
2023-04-01T11:31:27.008 WARNING:scrape:Not-understood crash indication 
2023-04-01T11:31:27.194 WARNING:scrape:Not-understood crash indication 2023-04-01T02:41:16.155 INFO:tasks.rgw.c1.client.0.smithi118.stderr:daemon-helper: command crashed with signal 6
2023-04-01T11:31:27.195 WARNING:scrape:Not-understood crash indication 2023-04-01T10:29:27.057 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi118.stderr: INFO: daemon-helper: command crashed with signal 6
2023-04-01T11:31:27.196 WARNING:scrape:Not-understood crash indication 
2023-04-01T11:31:27.388 WARNING:scrape:Not-understood crash indication 2023-04-01T02:41:16.155 INFO:tasks.rgw.c1.client.0.smithi118.stderr:daemon-helper: command crashed with signal 6
2023-04-01T11:31:27.389 WARNING:scrape:Not-understood crash indication 2023-04-01T10:29:27.057 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi118.stderr: INFO: daemon-helper: command crashed with signal 6
2023-04-01T11:31:27.390 WARNING:scrape:Not-understood crash indication 
2023-04-01T11:31:27.577 WARNING:scrape:Not-understood crash indication 2023-04-01T02:41:16.155 INFO:tasks.rgw.c1.client.0.smithi118.stderr:daemon-helper: command crashed with signal 6
2023-04-01T11:31:27.578 WARNING:scrape:Not-understood crash indication 2023-04-01T10:29:27.057 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi118.stderr: INFO: daemon-helper: command crashed with signal 6
2023-04-01T11:31:27.578 WARNING:scrape:Not-understood crash indication 
2023-04-01T11:31:28.471 WARNING:scrape:Not-understood crash indication 2023-04-01T01:38:41.781 INFO:tasks.rgw.c1.client.0.smithi043.stderr:daemon-helper: command crashed with signal 6
2023-04-01T11:31:28.472 WARNING:scrape:Not-understood crash indication 2023-04-01T07:44:29.777 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi043.stderr: INFO: daemon-helper: command crashed with signal 6
2023-04-01T11:31:28.473 WARNING:scrape:Not-understood crash indication 
2023-04-01T11:31:28.526 WARNING:scrape:Not-understood crash indication 2023-04-01T01:38:41.781 INFO:tasks.rgw.c1.client.0.smithi043.stderr:daemon-helper: command crashed with signal 6
2023-04-01T11:31:28.527 WARNING:scrape:Not-understood crash indication 2023-04-01T07:44:29.777 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi043.stderr: INFO: daemon-helper: command crashed with signal 6
2023-04-01T11:31:28.527 WARNING:scrape:Not-understood crash indication 
2023-04-01T11:31:28.932 INFO:scrape:Found 3 distinct failure reasons
2023-04-01T11:31:28.934 INFO:scrape:Failure: HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)')))
2023-04-01T11:31:28.934 INFO:scrape:20 jobs: ['7228833', '7228829', '7228813', '7228827', '7228809', '7228795', '7228847', '7228859', '7228789', '7228803', '7228819', '7228791', '7228845', '7228839', '7228799', '7228807', '7228855', '7228815', '7228821', '7228853']
2023-04-01T11:31:28.934 INFO:scrape:suites intersection: []
2023-04-01T11:31:28.935 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/testing', '2-kms/vault_old', '2-kms/vault_transit', '3-rgw/rgw', '4-tests/{s3tests}', 'frontend/beast', 'hadoop/default', 'hadoop/v32', 'http', 'ignore-pg-availability', 'install', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/dbstore/{cluster', 'rgw/hadoop-s3a/{clusters/fixed-2', 'rgw/multifs/{clusters/fixed-2', 'rgw/service-token/{clusters/fixed-1', 'rgw/tempest/{clusters/fixed-1', 'rgw/thrash/{clusters/fixed-2', 'rgw/website/{clusters/fixed-2', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3a-hadoop}', 's3tests-branch', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/rgw_bucket_quota', 'tasks/rgw_multipart_upload', 'tasks/rgw_ragweed', 'tasks/rgw_s3tests', 'tasks/rgw_s3tests}', 'tasks/rgw_tempest', 'tasks/rgw_user_quota', 'tasks/s3tests-website', 'tasks/service-token', 'thrasher/default', 'thrashosds-health', 'ubuntu_latest', 'ubuntu_latest}', 'workload/rgw_bucket_quota}', 'workload/rgw_multipart_upload}', 'workload/rgw_s3tests}', 'workload/rgw_user_quota}']
2023-04-01T11:31:28.935 INFO:scrape:
2023-04-01T11:31:28.935 INFO:scrape:Failure: rgw multisite test failures
2023-04-01T11:31:28.936 INFO:scrape:2 jobs: ['7228835', '7228801']
2023-04-01T11:31:28.936 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-04-01T11:31:28.936 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}']
2023-04-01T11:31:28.936 INFO:scrape:
2023-04-01T11:31:28.937 INFO:scrape:Failure: Cannot create secret
2023-04-01T11:31:28.937 INFO:scrape:1 jobs: ['7228825']
2023-04-01T11:31:28.937 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'supported-random-distro$/{ubuntu_20.04}}']
2023-04-01T11:31:28.938 INFO:scrape:
2023-04-01T11:31:28.940 INFO:teuthology.results:Sending results to yweinste@radhat.com: Test Run: yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/
failed:  23
dead:    0
running: 0
waiting: 0
queued:  0
passed:  14


Fail
=================================================================
[7228789]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:07:39
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228789/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228789/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ee566805efe74fb2bec6fdd76af4f9e4

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228791]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:36
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228791/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228791/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9a772a7d2a6c4e07a82ea61cc3613ca8

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228795]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:36
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228795/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228795/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3f5794d4a20b4e7aaccf5edfb92f6cd4

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228799]  rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
-----------------------------------------------------------------
time:   00:07:33
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228799/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228799/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6808ef1c1e0d4d1a883e3a8348b42f88

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228801]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}
-----------------------------------------------------------------
time:   07:45:56
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228801/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228801/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ed88b64cf34248b1a890491bcba8fa64

    rgw multisite test failures

[7228803]  rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
-----------------------------------------------------------------
time:   00:07:34
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228803/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228803/
sentry: https://sentry.ceph.com/organizations/ceph/?query=edb9d1a1805243e69d29975b7c65189a

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228807]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:35
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228807/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228807/
sentry: https://sentry.ceph.com/organizations/ceph/?query=088ad11885fc46df8a35a7b44797e5be

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228809]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:07:33
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228809/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228809/
sentry: https://sentry.ceph.com/organizations/ceph/?query=99e052c50f4e44268658d6911dc7e54a

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228813]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:41
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228813/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228813/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a89a812585104351a3762086517a1c8d

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228815]  rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
-----------------------------------------------------------------
time:   00:07:46
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228815/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228815/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d491e0233ef3420d9986288def411159

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228819]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:07:47
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228819/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228819/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3c653b167f684f3dad896460f676537b

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228821]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:46
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228821/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228821/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5389aa54373743879c03ff99d48111f9

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228825]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
-----------------------------------------------------------------
time:   00:16:08
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228825/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228825/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8a83c0eeec7f4e86ad2cd55189464d94

    Cannot create secret

[7228827]  rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests}
-----------------------------------------------------------------
time:   00:07:32
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228827/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228827/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fd17e404e30e439e9e568aecb82bc4e5

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228829]  rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
-----------------------------------------------------------------
time:   00:07:42
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228829/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228829/
sentry: https://sentry.ceph.com/organizations/ceph/?query=abdff462217547b891fcf1cf5b4eb18c

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228833]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:36
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228833/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228833/
sentry: https://sentry.ceph.com/organizations/ceph/?query=10e77de21baf487789d883fee066b209

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228835]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi}
-----------------------------------------------------------------
time:   10:11:53
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228835/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228835/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c937374e3a3a4545b637577229d322bb

    rgw multisite test failures

[7228839]  rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:41
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228839/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228839/
sentry: https://sentry.ceph.com/organizations/ceph/?query=42f9a8e7c9494ca19770e658e01c4cba

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228845]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:34
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228845/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228845/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b0457b8d76b34631bde796c3d233727d

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228847]  rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
-----------------------------------------------------------------
time:   00:07:48
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228847/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228847/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8117c3684d884f959aa9f7df3bea4158

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228853]  rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:39
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228853/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228853/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fe32c3f6a07e427ca3d37d175f08a47b

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228855]  rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:45
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228855/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228855/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f0e5b8a07fc642bdb3e267989507a166

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))

[7228859]  rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
-----------------------------------------------------------------
time:   00:07:45
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228859/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228859/
sentry: https://sentry.ceph.com/organizations/ceph/?query=da5f4106a90d4af7aec640f3cfc5d75a

    HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries
    exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4
    d/ubuntu/jammy/flavors/default/repo (Caused by
    SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: certificate has expired (_ssl.c:1108)')))



Pass
=================================================================
[7228787] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:57:55
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228787/

[7228793] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}}
time:   00:14:18
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228793/

[7228797] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_20.04}}
time:   00:14:38
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228797/

[7228805] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:34:08
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228805/

[7228811] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:15:44
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228811/

[7228817] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
time:   00:49:13
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228817/

[7228823] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/cloud_transition_s3tests}
time:   00:31:09
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228823/

[7228831] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/rgw_s3tests}
time:   00:24:31
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228831/

[7228837] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:54:38
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228837/

[7228841] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}}
time:   00:16:17
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228841/

[7228843] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{rhel_8} tasks/{0-install first ststests}}
time:   00:31:46
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228843/

[7228849] rgw/tools/{centos_latest cluster tasks}
time:   02:47:17
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228849/

[7228851] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:34:05
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228851/

[7228857] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:18:44
info:   http://pulpito.ceph.com/yuriw-2023-03-31_20:18:22-rgw-reef-distro-default-smithi/7228857/