2023-11-09T05:41:22.173 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-09T05:41:22.174 INFO:teuthology.results:seed: '7459'
2023-11-09T06:21:22.203 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-09T06:21:22.206 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-09T08:50:32.636 INFO:teuthology.results:Tests finished! gathering results...
2023-11-09T08:50:37.581 INFO:scrape:Found 70 jobs
2023-11-09T08:50:37.628 INFO:scrape:Found 2 distinct failure reasons
2023-11-09T08:50:37.629 INFO:scrape:Failure: rgw multisite test failures
2023-11-09T08:50:37.629 INFO:scrape:2 jobs: ['7452513', '7452549']
2023-11-09T08:50:37.630 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-11-09T08:50:37.630 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-11-09T08:50:37.631 INFO:scrape:
2023-11-09T08:50:37.631 INFO:scrape:Failure: Command failed on smithi100 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev'
2023-11-09T08:50:37.632 INFO:scrape:1 jobs: ['7452529']
2023-11-09T08:50:37.632 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}']
2023-11-09T08:50:37.633 INFO:scrape:
2023-11-09T08:50:37.636 INFO:teuthology.results:Sending results to gsalomon@redhat.com: Test Run: gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/
failed:  3
dead:    0
running: 0
waiting: 0
queued:  0
passed:  67


Fail
=================================================================
[7452513]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:56:05
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452513/
log:    http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452513/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6c91e2b84a5347afab335a3874db798e

    rgw multisite test failures

[7452529]  rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
-----------------------------------------------------------------
time:   00:14:42
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452529/
log:    http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452529/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ef5621b92a2c49f294b066863d8dd3bf

    Command failed on smithi100 with status 100:
    'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install
    python-dev'

[7452549]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:55:56
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452549/
log:    http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452549/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5c8507b769e14b4c824a306da8199fb7

    rgw multisite test failures



Pass
=================================================================
[7452506] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/cloud_transition_s3tests}
time:   00:31:57
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452506/

[7452507] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:39:03
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452507/

[7452508] rgw/d4n/{cluster ignore-pg-availability overrides supported-random-distro$/{centos_latest} tasks/rgw_d4ntests}
time:   00:15:17
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452508/

[7452509] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests}
time:   00:20:27
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452509/

[7452510] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop}
time:   00:26:03
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452510/

[7452511] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests}
time:   00:40:04
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452511/

[7452512] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:22:51
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452512/

[7452514] rgw/notifications/{beast bluestore-bitmap centos_latest fixed-2 ignore-pg-availability overrides tasks/{0-install test_amqp test_kafka test_others}}
time:   00:40:47
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452514/

[7452515] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest}
time:   00:18:13
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452515/

[7452516] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_latest}}
time:   00:19:35
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452516/

[7452517] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_latest} tasks/{0-install 1-keycloak 2-s3tests}}
time:   00:39:48
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452517/

[7452518] rgw/tempest/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/rgw_tempest ubuntu_latest}
time:   00:29:10
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452518/

[7452519] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:21:07
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452519/

[7452520] rgw/tools/{centos_latest cluster ignore-pg-availability tasks}
time:   02:42:09
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452520/

[7452521] rgw/upgrade/{1-install/pacific/{distro$/{centos_8.stream} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:24:32
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452521/

[7452522] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:47:39
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452522/

[7452523] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:25:59
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452523/

[7452524] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:49:41
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452524/

[7452525] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:21:09
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452525/

[7452526] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:37:14
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452526/

[7452527] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:52:33
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452527/

[7452528] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:21:33
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452528/

[7452530] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:46:35
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452530/

[7452531] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:27:44
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452531/

[7452532] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:09:49
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452532/

[7452533] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:22:42
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452533/

[7452534] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:17:57
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452534/

[7452535] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:52:19
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452535/

[7452536] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:21:25
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452536/

[7452537] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:23:50
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452537/

[7452538] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:31:08
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452538/

[7452539] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:20:27
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452539/

[7452540] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_latest}}
time:   00:19:23
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452540/

[7452541] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:39:57
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452541/

[7452542] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:55:43
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452542/

[7452543] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:20:42
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452543/

[7452544] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:51:54
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452544/

[7452545] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:56:35
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452545/

[7452546] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:23:09
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452546/

[7452547] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:30:08
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452547/

[7452548] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop}
time:   00:23:40
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452548/

[7452550] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:27:12
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452550/

[7452551] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:20:14
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452551/

[7452552] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:50:18
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452552/

[7452553] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:29:04
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452553/

[7452554] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:54:06
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452554/

[7452555] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:50:26
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452555/

[7452556] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:21:06
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452556/

[7452557] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:46:40
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452557/

[7452558] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:31:33
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452558/

[7452559] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:21:33
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452559/

[7452560] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_latest}}
time:   00:16:53
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452560/

[7452561] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:36:38
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452561/

[7452562] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:43:42
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452562/

[7452563] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:16:02
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452563/

[7452564] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:18:45
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452564/

[7452565] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides}
time:   00:17:24
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452565/

[7452566] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:39:52
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452566/

[7452567] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:54:32
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452567/

[7452568] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:18:11
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452568/

[7452569] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest}
time:   00:26:36
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452569/

[7452570] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:48:00
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452570/

[7452571] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:24:21
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452571/

[7452572] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   00:51:46
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452572/

[7452573] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}
time:   00:49:26
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452573/

[7452574] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:16:05
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452574/

[7452575] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind}
time:   01:17:02
info:   https://pulpito.ceph.com/gsalomon-2023-11-09_05:41:08-rgw-exception_cause_crash-distro-default-smithi/7452575/