2023-12-12T22:22:48.108 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-12T22:22:48.108 INFO:teuthology.results:seed: '9490' 2023-12-13T03:49:48.264 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-13T03:49:48.289 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-12-13T05:13:54.261 INFO:teuthology.results:Tests finished! gathering results... 2023-12-13T05:13:59.762 INFO:scrape:Found 72 jobs 2023-12-13T05:14:00.916 WARNING:scrape:Not-understood crash indication 2023-12-12T23:25:41.848 INFO:tasks.rgw.c1.client.0.smithi116.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:00.916 WARNING:scrape:Not-understood crash indication 2023-12-12T23:25:41.882 INFO:tasks.rgw.c1.client.1.smithi116.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:00.916 WARNING:scrape:Not-understood crash indication 2023-12-13T00:49:49.853 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi116.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:00.916 WARNING:scrape:Not-understood crash indication 2023-12-13T00:49:49.853 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi116.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:00.916 WARNING:scrape:Not-understood crash indication 2023-12-13T05:14:01.126 WARNING:scrape:Not-understood crash indication 2023-12-12T23:25:41.848 INFO:tasks.rgw.c1.client.0.smithi116.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.126 WARNING:scrape:Not-understood crash indication 2023-12-12T23:25:41.882 INFO:tasks.rgw.c1.client.1.smithi116.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.126 WARNING:scrape:Not-understood crash indication 2023-12-13T00:49:49.853 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi116.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.127 WARNING:scrape:Not-understood crash indication 2023-12-13T00:49:49.853 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi116.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.127 WARNING:scrape:Not-understood crash indication 2023-12-13T05:14:01.333 WARNING:scrape:Not-understood crash indication 2023-12-12T23:25:41.848 INFO:tasks.rgw.c1.client.0.smithi116.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.333 WARNING:scrape:Not-understood crash indication 2023-12-12T23:25:41.882 INFO:tasks.rgw.c1.client.1.smithi116.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.334 WARNING:scrape:Not-understood crash indication 2023-12-13T00:49:49.853 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi116.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.334 WARNING:scrape:Not-understood crash indication 2023-12-13T00:49:49.853 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi116.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.334 WARNING:scrape:Not-understood crash indication 2023-12-13T05:14:01.544 WARNING:scrape:Not-understood crash indication 2023-12-12T23:25:41.848 INFO:tasks.rgw.c1.client.0.smithi116.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.544 WARNING:scrape:Not-understood crash indication 2023-12-12T23:25:41.882 INFO:tasks.rgw.c1.client.1.smithi116.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.545 WARNING:scrape:Not-understood crash indication 2023-12-13T00:49:49.853 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi116.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.545 WARNING:scrape:Not-understood crash indication 2023-12-13T00:49:49.853 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi116.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:01.545 WARNING:scrape:Not-understood crash indication 2023-12-13T05:14:02.186 WARNING:scrape:Not-understood crash indication 2023-12-12T22:52:22.391 INFO:tasks.rgw.c2.client.0.smithi196.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:02.186 WARNING:scrape:Not-understood crash indication 2023-12-12T22:52:42.267 INFO:tasks.rgw.c1.client.0.smithi121.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:02.187 WARNING:scrape:Not-understood crash indication 2023-12-13T00:28:07.859 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi196.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:02.187 WARNING:scrape:Not-understood crash indication 2023-12-13T00:28:07.866 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi121.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:02.187 WARNING:scrape:Not-understood crash indication 2023-12-13T05:14:02.209 WARNING:scrape:Not-understood crash indication 2023-12-12T22:52:22.391 INFO:tasks.rgw.c2.client.0.smithi196.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:02.209 WARNING:scrape:Not-understood crash indication 2023-12-12T22:52:42.267 INFO:tasks.rgw.c1.client.0.smithi121.stderr:daemon-helper: command crashed with signal 11 2023-12-13T05:14:02.209 WARNING:scrape:Not-understood crash indication 2023-12-13T00:28:07.859 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi196.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:02.209 WARNING:scrape:Not-understood crash indication 2023-12-13T00:28:07.866 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi121.stderr: INFO: daemon-helper: command crashed with signal 11 2023-12-13T05:14:02.209 WARNING:scrape:Not-understood crash indication 2023-12-13T05:14:03.300 INFO:scrape:Found 4 distinct failure reasons 2023-12-13T05:14:03.300 INFO:scrape:Failure: 2023-12-13T05:14:03.300 INFO:scrape:3 jobs: ['7489157', '7489181', '7489202'] 2023-12-13T05:14:03.300 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/singleton/{all/radosgw-admin', 'supported-random-distro$/{centos_latest}}'] 2023-12-13T05:14:03.300 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/singleton/{all/radosgw-admin', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 'supported-random-distro$/{centos_latest}}'] 2023-12-13T05:14:03.300 INFO:scrape: 2023-12-13T05:14:03.301 INFO:scrape:Failure: rgw multisite test failures 2023-12-13T05:14:03.301 INFO:scrape:2 jobs: ['7489190', '7489154'] 2023-12-13T05:14:03.301 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2023-12-13T05:14:03.301 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2023-12-13T05:14:03.301 INFO:scrape: 2023-12-13T05:14:03.301 INFO:scrape:Failure: Command failed on smithi138 with status 100: 'DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev' 2023-12-13T05:14:03.301 INFO:scrape:1 jobs: ['7489170'] 2023-12-13T05:14:03.301 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability', 'rgw/crypt/{0-cluster/fixed-1', 's3tests-branch', 'ubuntu_latest}'] 2023-12-13T05:14:03.301 INFO:scrape: 2023-12-13T05:14:03.301 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fc5db945edc] 2023-12-13T05:14:03.302 INFO:scrape:ceph version 19.0.0-101-gea6e3cb0 (ea6e3cb09c76e02a32497ec688d287d9ddd368bb) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc5da6f9520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fc5db945edc] 6: /usr/lib/ceph/libceph-common.so.2(+0x170036) [0x7fc5db946036] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55ded5d4aef1] 8: radosgw-admin(+0x858f0a) [0x55ded5d53f0a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55ded5d5854b] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc5da6e0d90] 12: __libc_start_main() 13: _start() 2023-12-13T05:14:03.302 INFO:scrape:1 jobs: ['7489204'] 2023-12-13T05:14:03.302 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'datacache/no_datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{bucket-check', 'validater/valgrind}', 'versioning}'] 2023-12-13T05:14:03.302 INFO:scrape: