2023-11-09T13:26:46.907 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T13:26:46.908 INFO:teuthology.results:seed: '1929' 2023-11-09T15:51:48.537 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T15:51:48.565 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-09T18:18:58.794 INFO:teuthology.results:Tests finished! gathering results... 2023-11-09T18:19:04.145 INFO:scrape:Found 70 jobs 2023-11-09T18:19:04.899 INFO:scrape:Found 5 distinct failure reasons 2023-11-09T18:19:04.900 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f18956aaef0] 2023-11-09T18:19:04.900 INFO:scrape:ceph version 18.0.0-7221-g5686a21b (5686a21b8a7ddf8584345bf973cc3353f14a68a5) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f189425d520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f18956aaef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f18956ab04a] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x5612df9f26d1] 8: radosgw-admin(+0x867ffa) [0x5612df9f9ffa] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x5612dfa003fb] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f1894244d90] 12: __libc_start_main() 13: _start() 2023-11-09T18:19:04.901 INFO:scrape:1 jobs: ['7452819'] 2023-11-09T18:19:04.901 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2023-11-09T18:19:04.901 INFO:scrape: 2023-11-09T18:19:04.902 INFO:scrape:Failure: Command failed on smithi084 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev' 2023-11-09T18:19:04.902 INFO:scrape:1 jobs: ['7452803'] 2023-11-09T18:19:04.903 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}'] 2023-11-09T18:19:04.903 INFO:scrape: 2023-11-09T18:19:04.904 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f565b96ef52] 2023-11-09T18:19:04.904 INFO:scrape:ceph version 18.0.0-7221-g5686a21b (5686a21b8a7ddf8584345bf973cc3353f14a68a5) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f5658e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f5658ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f565b96ef52] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f565b96f0b6] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55fb9b6a4381] 8: radosgw-admin(+0x7990c2) [0x55fb9b6b20c2] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55fb9b6ba6cb] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7f5658e3feb0] 12: __libc_start_main() 13: _start() 2023-11-09T18:19:04.904 INFO:scrape:1 jobs: ['7452849'] 2023-11-09T18:19:04.905 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2023-11-09T18:19:04.905 INFO:scrape: 2023-11-09T18:19:04.905 INFO:scrape:Failure: rgw multisite test failures 2023-11-09T18:19:04.906 INFO:scrape:2 jobs: ['7452823', '7452787'] 2023-11-09T18:19:04.906 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'tasks/test_multi}'] 2023-11-09T18:19:04.907 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'tasks/test_multi}'] 2023-11-09T18:19:04.907 INFO:scrape: 2023-11-09T18:19:04.907 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3c5aac5ef0] 2023-11-09T18:19:04.908 INFO:scrape:ceph version 18.0.0-7221-g5686a21b (5686a21b8a7ddf8584345bf973cc3353f14a68a5) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3c59678520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f3c5aac5ef0] 6: /usr/lib/ceph/libceph-common.so.2(+0x17004a) [0x7f3c5aac604a] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562e61df56d1] 8: radosgw-admin(+0x867ffa) [0x562e61dfcffa] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x562e61e033fb] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f3c5965fd90] 12: __libc_start_main() 13: _start() 2023-11-09T18:19:04.908 INFO:scrape:1 jobs: ['7452828'] 2023-11-09T18:19:04.908 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2023-11-09T18:19:04.909 INFO:scrape: 2023-11-09T18:19:04.912 INFO:teuthology.results:Sending results to gsalomon@redhat.com: Test Run: gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 64 Fail ================================================================= [7452787] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} ----------------------------------------------------------------- time: 04:39:01 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452787/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a02f473a4604cf399e46dd6cf110f5a rgw multisite test failures [7452803] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} ----------------------------------------------------------------- time: 00:13:27 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452803/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452803/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d5471c3196843b993f745663d594c34 Command failed on smithi084 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev' [7452819] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 01:27:42 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452819/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452819/ saw valgrind issues [7452823] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi} ----------------------------------------------------------------- time: 02:46:38 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452823/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452823/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aaf2200fe76c4e59b2c5cc822d1e8746 rgw multisite test failures [7452828] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 00:42:53 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452828/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452828/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7427de886e044594993f4479fbc6de95 Command failed on smithi113 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test' [7452849] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} ----------------------------------------------------------------- time: 01:07:39 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452849/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b0c85e374904a48a2f2240881bf7759 Command failed (s3 tests against rgw) on smithi046 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3'" Pass ================================================================= [7452780] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/cloud_transition_s3tests} time: 00:31:50 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452780/ [7452781] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:36:38 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452781/ [7452782] rgw/d4n/{cluster ignore-pg-availability overrides supported-random-distro$/{ubuntu_latest} tasks/rgw_d4ntests} time: 00:17:56 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452782/ [7452783] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests} time: 00:19:53 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452783/ [7452784] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop} time: 00:21:21 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452784/ [7452785] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests} time: 00:37:50 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452785/ [7452786] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:17:53 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452786/ [7452788] rgw/notifications/{beast bluestore-bitmap centos_latest fixed-2 ignore-pg-availability overrides tasks/{0-install test_amqp test_kafka test_others}} time: 00:41:53 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452788/ [7452789] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest} time: 00:19:09 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452789/ [7452790] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_latest}} time: 00:17:53 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452790/ [7452791] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install 1-keycloak 2-s3tests}} time: 00:43:17 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452791/ [7452792] rgw/tempest/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/rgw_tempest ubuntu_latest} time: 00:25:43 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452792/ [7452793] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:20:39 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452793/ [7452794] rgw/tools/{centos_latest cluster ignore-pg-availability tasks} time: 02:46:21 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452794/ [7452795] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:19:54 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452795/ [7452796] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:37:22 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452796/ [7452797] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest} time: 00:26:03 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452797/ [7452798] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:06:13 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452798/ [7452799] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:20:04 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452799/ [7452800] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:37:45 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452800/ [7452801] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:46:55 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452801/ [7452802] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:20:46 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452802/ [7452804] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:47:05 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452804/ [7452805] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:27:58 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452805/ [7452806] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:07:18 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452806/ [7452807] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:19:14 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452807/ [7452808] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:19:22 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452808/ [7452809] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:40:17 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452809/ [7452810] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:19:26 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452810/ [7452811] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:20:00 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452811/ [7452812] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:29:19 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452812/ [7452813] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:19:34 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452813/ [7452814] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_latest}} time: 00:18:17 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452814/ [7452815] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:50:00 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452815/ [7452816] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 00:53:46 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452816/ [7452817] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:20:13 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452817/ [7452818] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:50:42 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452818/ [7452820] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:21:21 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452820/ [7452821] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:28:42 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452821/ [7452822] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop} time: 00:22:53 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452822/ [7452824] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:28:29 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452824/ [7452825] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:18:14 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452825/ [7452826] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:43:44 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452826/ [7452827] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:24:33 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452827/ [7452829] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:44:51 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452829/ [7452830] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:15:56 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452830/ [7452831] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:05:45 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452831/ [7452832] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:26:52 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452832/ [7452833] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:16:35 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452833/ [7452834] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_latest}} time: 00:16:13 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452834/ [7452835] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:44:38 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452835/ [7452836] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:06:54 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452836/ [7452837] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:16:02 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452837/ [7452838] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:16:48 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452838/ [7452839] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:17:15 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452839/ [7452840] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:39:08 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452840/ [7452841] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 00:52:12 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452841/ [7452842] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:17:29 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452842/ [7452843] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:25:54 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452843/ [7452844] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:48:24 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452844/ [7452845] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:24:26 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452845/ [7452846] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 00:51:18 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452846/ [7452847] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:48:03 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452847/ [7452848] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:16:00 info: https://pulpito.ceph.com/gsalomon-2023-11-09_13:26:30-rgw-exception_cause_crash-distro-default-smithi/7452848/