Found 70 jobs
Missing teuthology log /home/teuthworker/archive/dang-2023-09-06_10:06:10-rgw-wip-dang-posix-driver-distro-default-smithi/7390170/teuthology.log
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Not-understood crash indication 2023-09-06T17:38:35.121 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11
Not-understood crash indication 2023-09-06T17:57:37.857 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11
Not-understood crash indication 2023-09-06T18:37:48.460 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi033.stderr: INFO: daemon-helper: command crashed with signal 11
Not-understood crash indication 
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Saw end of BT but not start
Not-understood crash indication 2023-09-06T17:38:35.121 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11
Not-understood crash indication 2023-09-06T17:57:37.857 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11
Not-understood crash indication 2023-09-06T18:37:48.460 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi033.stderr: INFO: daemon-helper: command crashed with signal 11
Not-understood crash indication 
Found 6 distinct failure reasons
Failure: Command failed on smithi005 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev'
1 jobs: ['7390137']
suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}']

Dead: None
1 jobs: ['7390170']

Failure: rgw multisite test failures
2 jobs: ['7390121', '7390157']
suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/test_multi}']

Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f2a879631bd]
ceph version 18.0.0-5781-g284a31c6 (284a31c6afd1841dcbf6a39d2f01a5d8c37f876a) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f2a84e54db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f2a84ea154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f2a879631bd]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x163321) [0x7f2a87963321]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x560ffed28131]
 8: radosgw-admin(+0x782fa2) [0x560ffed35fa2]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x560ffed3e20b]
 10: main()
 11: /lib64/libc.so.6(+0x3feb0) [0x7f2a84e3feb0]
 12: __libc_start_main()
 13: _start()
1 jobs: ['7390153']
suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']

Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f45ddd631bd]
ceph version 18.0.0-5781-g284a31c6 (284a31c6afd1841dcbf6a39d2f01a5d8c37f876a) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f45db254db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f45db2a154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f45ddd631bd]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x163321) [0x7f45ddd63321]
 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> (*)(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x55f386b66131]
 8: radosgw-admin(+0x782fa2) [0x55f386b73fa2]
 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55f386b7c20b]
 10: main()
 11: /lib64/libc.so.6(+0x3feb0) [0x7f45db23feb0]
 12: __libc_start_main()
 13: _start()
1 jobs: ['7390140']
suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}']

Failure: Command failed (s3 tests against rgw) on smithi029 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and not fails_on_dbstore'"
1 jobs: ['7390117']
suites: ['ignore-pg-availability', 'overrides', 'rgw/dbstore/{cluster', 's3tests-branch', 'supported-random-distro$/{centos_latest}', 'tasks/rgw_s3tests}']