2023-09-06T15:32:10.129 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-06T15:32:10.130 INFO:teuthology.results:seed: '375' 2023-09-06T17:00:40.056 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-06T17:00:40.087 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-06T19:29:51.506 INFO:teuthology.results:Tests finished! gathering results... 2023-09-06T19:29:56.598 INFO:scrape:Found 70 jobs 2023-09-06T19:29:56.733 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/dang-2023-09-06_10:06:10-rgw-wip-dang-posix-driver-distro-default-smithi/7390170/teuthology.log 2023-09-06T19:29:57.395 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.396 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.396 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.397 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.397 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.398 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.418 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.418 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.419 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.419 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.450 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.451 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.451 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.452 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.813 WARNING:scrape:Not-understood crash indication 2023-09-06T17:38:35.121 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11 2023-09-06T19:29:57.814 WARNING:scrape:Not-understood crash indication 2023-09-06T17:57:37.857 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11 2023-09-06T19:29:57.815 WARNING:scrape:Not-understood crash indication 2023-09-06T18:37:48.460 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi033.stderr: INFO: daemon-helper: command crashed with signal 11 2023-09-06T19:29:57.815 WARNING:scrape:Not-understood crash indication 2023-09-06T19:29:57.948 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.949 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.949 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.949 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.950 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.950 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.967 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.967 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.968 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.968 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.981 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.982 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.982 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:57.983 WARNING:scrape:Saw end of BT but not start 2023-09-06T19:29:58.225 WARNING:scrape:Not-understood crash indication 2023-09-06T17:38:35.121 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11 2023-09-06T19:29:58.226 WARNING:scrape:Not-understood crash indication 2023-09-06T17:57:37.857 INFO:tasks.rgw.c1.client.1.smithi033.stderr:daemon-helper: command crashed with signal 11 2023-09-06T19:29:58.227 WARNING:scrape:Not-understood crash indication 2023-09-06T18:37:48.460 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi033.stderr: INFO: daemon-helper: command crashed with signal 11 2023-09-06T19:29:58.227 WARNING:scrape:Not-understood crash indication 2023-09-06T19:29:58.854 INFO:scrape:Found 6 distinct failure reasons 2023-09-06T19:29:58.855 INFO:scrape:Failure: Command failed on smithi005 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev' 2023-09-06T19:29:58.856 INFO:scrape:1 jobs: ['7390137'] 2023-09-06T19:29:58.856 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}'] 2023-09-06T19:29:58.857 INFO:scrape: 2023-09-06T19:29:58.857 INFO:scrape:Dead: None 2023-09-06T19:29:58.857 INFO:scrape:1 jobs: ['7390170'] 2023-09-06T19:29:58.858 INFO:scrape: 2023-09-06T19:29:58.858 INFO:scrape:Failure: rgw multisite test failures 2023-09-06T19:29:58.859 INFO:scrape:2 jobs: ['7390121', '7390157'] 2023-09-06T19:29:58.859 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2023-09-06T19:29:58.859 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/test_multi}'] 2023-09-06T19:29:58.860 INFO:scrape: 2023-09-06T19:29:58.860 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f2a879631bd] 2023-09-06T19:29:58.861 INFO:scrape:ceph version 18.0.0-5781-g284a31c6 (284a31c6afd1841dcbf6a39d2f01a5d8c37f876a) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f2a84e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f2a84ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f2a879631bd] 6: /usr/lib64/ceph/libceph-common.so.2(+0x163321) [0x7f2a87963321] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x560ffed28131] 8: radosgw-admin(+0x782fa2) [0x560ffed35fa2] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x560ffed3e20b] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7f2a84e3feb0] 12: __libc_start_main() 13: _start() 2023-09-06T19:29:58.861 INFO:scrape:1 jobs: ['7390153'] 2023-09-06T19:29:58.861 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}'] 2023-09-06T19:29:58.862 INFO:scrape: 2023-09-06T19:29:58.862 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f45ddd631bd] 2023-09-06T19:29:58.863 INFO:scrape:ceph version 18.0.0-5781-g284a31c6 (284a31c6afd1841dcbf6a39d2f01a5d8c37f876a) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f45db254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f45db2a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f45ddd631bd] 6: /usr/lib64/ceph/libceph-common.so.2(+0x163321) [0x7f45ddd63321] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55f386b66131] 8: radosgw-admin(+0x782fa2) [0x55f386b73fa2] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55f386b7c20b] 10: main() 11: /lib64/libc.so.6(+0x3feb0) [0x7f45db23feb0] 12: __libc_start_main() 13: _start() 2023-09-06T19:29:58.864 INFO:scrape:1 jobs: ['7390140'] 2023-09-06T19:29:58.864 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{centos_latest}', 'tasks/{cls', 'validater/valgrind}', 'versioning}'] 2023-09-06T19:29:58.864 INFO:scrape: 2023-09-06T19:29:58.865 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi029 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and not fails_on_dbstore'" 2023-09-06T19:29:58.865 INFO:scrape:1 jobs: ['7390117'] 2023-09-06T19:29:58.866 INFO:scrape:suites: ['ignore-pg-availability', 'overrides', 'rgw/dbstore/{cluster', 's3tests-branch', 'supported-random-distro$/{centos_latest}', 'tasks/rgw_s3tests}'] 2023-09-06T19:29:58.866 INFO:scrape: