Found 49 jobs Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Not-understood crash indication 2023-09-21T04:24:57.633 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-09-21T05:05:45.153 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi033.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Not-understood crash indication 2023-09-21T04:24:57.633 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-09-21T05:05:45.153 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi033.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Not-understood crash indication 2023-09-21T04:24:57.633 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-09-21T05:05:45.153 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi033.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Not-understood crash indication 2023-09-21T04:24:57.633 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-09-21T05:05:45.153 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi033.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Not-understood crash indication 2023-09-21T04:24:57.633 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication 2023-09-21T05:05:45.153 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi033.stderr: INFO: daemon-helper: command crashed with signal 11 Not-understood crash indication Found 4 distinct failure reasons Failure: Command failed (s3 tests against rgw) on smithi098 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and not fails_on_dbstore'" 2 jobs: ['7399638', '7399640'] suites intersection: ['ignore-pg-availability', 'overrides', 's3tests-branch', 'tasks/rgw_s3tests}'] suites union: ['ignore-pg-availability', 'overrides', 'rgw/dbstore/{cluster', 'rgw/lifecycle/{cluster', 's3tests-branch', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/rgw_s3tests}'] Failure: Command failed on smithi145 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev' 1 jobs: ['7399655'] suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}'] Failure: rgw multisite test failures 2 jobs: ['7399668', '7399642'] suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fe627f848dd] ceph version 18.0.0-6197-g24d95fda (24d95fda2306160690975bb971c235a0e68ea9dc) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fe626b20520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fe627f848dd] 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7fe627f84a37] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a1fdb73631] 8: radosgw-admin(+0x857c3a) [0x55a1fdb7ac3a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a1fdb8129b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fe626b07d90] 12: __libc_start_main() 13: _start() 1 jobs: ['7399663'] suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']