Found 12 jobs Ignoring malparsed backtrace: ceph version 18.0.0-1783-g9ffcb1a0 (9ffcb1a0ee407ff1fc15150c7147dce6acc92e84) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcaa0ca8ea0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5650f81bc0ca] Ignoring malparsed backtrace: 2023-01-13T10:18:25.551+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.558+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.755 INFO:teuthology.orchestra.run.smithi101.stdout: Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/ceph.py", line 1921, in task , yield , File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/run_tasks.py", line 188, in run_tasks Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/rados.py", line 286, in task , running.get() Ignoring malparsed backtrace: 2023-01-13T10:19:54.470 INFO:teuthology.orchestra.run.smithi101.stdout:/usr/bin/ceph_test_rados , 2023-01-13T10:19:54.470 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): Ignoring malparsed backtrace: ceph version 18.0.0-1783-g9ffcb1a0 (9ffcb1a0ee407ff1fc15150c7147dce6acc92e84) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcaa0ca8ea0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5650f81bc0ca] Ignoring malparsed backtrace: 2023-01-13T10:18:25.551+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.558+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.755 INFO:teuthology.orchestra.run.smithi101.stdout: Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/ceph.py", line 1921, in task , yield , File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/run_tasks.py", line 188, in run_tasks Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/rados.py", line 286, in task , running.get() Ignoring malparsed backtrace: 2023-01-13T10:19:54.470 INFO:teuthology.orchestra.run.smithi101.stdout:/usr/bin/ceph_test_rados , 2023-01-13T10:19:54.470 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): Ignoring malparsed backtrace: ceph version 18.0.0-1783-g9ffcb1a0 (9ffcb1a0ee407ff1fc15150c7147dce6acc92e84) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcaa0ca8ea0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5650f81bc0ca] Ignoring malparsed backtrace: 2023-01-13T10:18:25.551+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.558+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.755 INFO:teuthology.orchestra.run.smithi101.stdout: Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/ceph.py", line 1921, in task , yield , File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/run_tasks.py", line 188, in run_tasks Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/rados.py", line 286, in task , running.get() Ignoring malparsed backtrace: 2023-01-13T10:19:54.470 INFO:teuthology.orchestra.run.smithi101.stdout:/usr/bin/ceph_test_rados , 2023-01-13T10:19:54.470 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): Ignoring malparsed backtrace: ceph version 18.0.0-1783-g9ffcb1a0 (9ffcb1a0ee407ff1fc15150c7147dce6acc92e84) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcaa0ca8ea0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5650f81bc0ca] Ignoring malparsed backtrace: 2023-01-13T10:18:25.551+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.558+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.755 INFO:teuthology.orchestra.run.smithi101.stdout: Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/ceph.py", line 1921, in task , yield , File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/run_tasks.py", line 188, in run_tasks Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/rados.py", line 286, in task , running.get() Ignoring malparsed backtrace: 2023-01-13T10:19:54.470 INFO:teuthology.orchestra.run.smithi101.stdout:/usr/bin/ceph_test_rados , 2023-01-13T10:19:54.470 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): Ignoring malparsed backtrace: ceph version 18.0.0-1783-g9ffcb1a0 (9ffcb1a0ee407ff1fc15150c7147dce6acc92e84) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcaa0ca8ea0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5650f81bc0ca] Ignoring malparsed backtrace: 2023-01-13T10:18:25.551+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.558+0000 7f876efcc700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:25.755 INFO:teuthology.orchestra.run.smithi101.stdout: Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/ceph.py", line 1921, in task , yield , File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/run_tasks.py", line 188, in run_tasks Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_9ffcb1a0ee407ff1fc15150c7147dce6acc92e84/qa/tasks/rados.py", line 286, in task , running.get() Ignoring malparsed backtrace: 2023-01-13T10:19:54.470 INFO:teuthology.orchestra.run.smithi101.stdout:/usr/bin/ceph_test_rados , 2023-01-13T10:19:54.470 ERROR:teuthology.run_tasks:Manager failed: internal.archive , Traceback (most recent call last): Ignoring malparsed backtrace: ceph version 18.0.0-1783-g9ffcb1a0 (9ffcb1a0ee407ff1fc15150c7147dce6acc92e84) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f15088e9ea0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55d37fe220ca] Ignoring malparsed backtrace: 2023-01-13T10:18:32.189 INFO:teuthology.orchestra.run.smithi077.stdout: , 2023-01-13T10:18:32.190 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":136,"fsid":"cb60268d-e37a-4a7a-85fb-ee1f96a8d263","created":"2023-01-13T09:47:58.104752+0000","modified":"2023-01-13T10:18:31.513310+0000","last_up_change":"2023-01-13T09:48:06.088475+0000","last_in_change":"2023-01-13T09:48:00.445778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-13T09:48:07.779681+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-13T09:48:08.702557+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-01-13T09:48:15.560751+0000","flags":262305,"flags_names":"hashpspool,nopgchange,write_fadvise_dontneed,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"b348d66e-ea99-4059-a761-8218d83a739b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":101993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6801","nonce":101993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":101993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6805","nonce":101993}]},"public_addr":"172.21.15.77:6800/101993","cluster_addr":"172.21.15.77:6801/101993","heartbeat_back_addr":"172.21.15.77:6804/101993","heartbeat_front_addr":"172.21.15.77:6805/101993","state":["exists","up"]},{"osd":1,"uuid":"80963b8f-6919-4dc0-8c42-cb45a8d27295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6803","nonce":102020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":102020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":102020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":102020}]},"public_addr":"172.21.15.77:6803/102020","cluster_addr":"172.21.15.77:6802/102020","heartbeat_back_addr":"172.21.15.77:6807/102020","heartbeat_front_addr":"172.21.15.77:6806/102020","state":["exists","up"]},{"osd":2,"uuid":"aa19a7b7-3a48-482e-93a6-5abe400f210c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":101672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":101672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":101672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":101672}]},"public_addr":"172.21.15.93:6803/101672","cluster_addr":"172.21.15.93:6802/101672","heartbeat_back_addr":"172.21.15.93:6806/101672","heartbeat_front_addr":"172.21.15.93:6807/101672","state":["exists","up"]},{"osd":3,"uuid":"0408e7fb-f03e-42c0-ad91-ce0a90e7d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":101700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":101700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6809","nonce":101700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":101700}]},"public_addr":"172.21.15.93:6805/101700","cluster_addr":"172.21.15.93:6804/101700","heartbeat_back_addr":"172.21.15.93:6809/101700","heartbeat_front_addr":"172.21.15.93:6808/101700","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2023-01-13T10:18:32.205 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats Ignoring malparsed backtrace: 2023-01-13T10:18:36.750 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-01-13T10:18:36.871+0000 7f82fee44700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:36.877+0000 7f82fee44700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-01-13T10:19:13.541 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/log/ceph -- . , 2023-01-13T10:19:17.961 DEBUG:teuthology.misc:Transferring archived files from smithi093:/var/log/ceph to /home/teuthworker/archive/sjust-2023-01-13_09:24:42-crimson-rados-wip-sjust-testing-2023-01-12-223717-distro-default-smithi/7141074/remote/smithi093/log , 2023-01-13T10:19:17.963 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar cz -f - -C /var/log/ceph -- . Ignoring malparsed backtrace: 2023-01-13T10:19:43.296 INFO:teuthology.task.internal: core looks like: /tmp/tmp14qlen2z: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-', real uid: 1000, effective uid: 1000, real gid: 1268, effective gid: 1268, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64' , , 2023-01-13T10:19:43.347 INFO:teuthology.task.internal: dump_program: ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in- Ignoring malparsed backtrace: ceph version 18.0.0-1783-g9ffcb1a0 (9ffcb1a0ee407ff1fc15150c7147dce6acc92e84) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f15088e9ea0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55d37fe220ca] Ignoring malparsed backtrace: 2023-01-13T10:18:32.189 INFO:teuthology.orchestra.run.smithi077.stdout: , 2023-01-13T10:18:32.190 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":136,"fsid":"cb60268d-e37a-4a7a-85fb-ee1f96a8d263","created":"2023-01-13T09:47:58.104752+0000","modified":"2023-01-13T10:18:31.513310+0000","last_up_change":"2023-01-13T09:48:06.088475+0000","last_in_change":"2023-01-13T09:48:00.445778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-13T09:48:07.779681+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-13T09:48:08.702557+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-01-13T09:48:15.560751+0000","flags":262305,"flags_names":"hashpspool,nopgchange,write_fadvise_dontneed,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"b348d66e-ea99-4059-a761-8218d83a739b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":101993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6801","nonce":101993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":101993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6805","nonce":101993}]},"public_addr":"172.21.15.77:6800/101993","cluster_addr":"172.21.15.77:6801/101993","heartbeat_back_addr":"172.21.15.77:6804/101993","heartbeat_front_addr":"172.21.15.77:6805/101993","state":["exists","up"]},{"osd":1,"uuid":"80963b8f-6919-4dc0-8c42-cb45a8d27295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6803","nonce":102020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":102020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":102020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":102020}]},"public_addr":"172.21.15.77:6803/102020","cluster_addr":"172.21.15.77:6802/102020","heartbeat_back_addr":"172.21.15.77:6807/102020","heartbeat_front_addr":"172.21.15.77:6806/102020","state":["exists","up"]},{"osd":2,"uuid":"aa19a7b7-3a48-482e-93a6-5abe400f210c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":101672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":101672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":101672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":101672}]},"public_addr":"172.21.15.93:6803/101672","cluster_addr":"172.21.15.93:6802/101672","heartbeat_back_addr":"172.21.15.93:6806/101672","heartbeat_front_addr":"172.21.15.93:6807/101672","state":["exists","up"]},{"osd":3,"uuid":"0408e7fb-f03e-42c0-ad91-ce0a90e7d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":101700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":101700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6809","nonce":101700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":101700}]},"public_addr":"172.21.15.93:6805/101700","cluster_addr":"172.21.15.93:6804/101700","heartbeat_back_addr":"172.21.15.93:6809/101700","heartbeat_front_addr":"172.21.15.93:6808/101700","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2023-01-13T10:18:32.205 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats Ignoring malparsed backtrace: 2023-01-13T10:18:36.750 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-01-13T10:18:36.871+0000 7f82fee44700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:36.877+0000 7f82fee44700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-01-13T10:19:13.541 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/log/ceph -- . , 2023-01-13T10:19:17.961 DEBUG:teuthology.misc:Transferring archived files from smithi093:/var/log/ceph to /home/teuthworker/archive/sjust-2023-01-13_09:24:42-crimson-rados-wip-sjust-testing-2023-01-12-223717-distro-default-smithi/7141074/remote/smithi093/log , 2023-01-13T10:19:17.963 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar cz -f - -C /var/log/ceph -- . Ignoring malparsed backtrace: 2023-01-13T10:19:43.296 INFO:teuthology.task.internal: core looks like: /tmp/tmp14qlen2z: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-', real uid: 1000, effective uid: 1000, real gid: 1268, effective gid: 1268, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64' , , 2023-01-13T10:19:43.347 INFO:teuthology.task.internal: dump_program: ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in- Ignoring malparsed backtrace: ceph version 18.0.0-1783-g9ffcb1a0 (9ffcb1a0ee407ff1fc15150c7147dce6acc92e84) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f15088e9ea0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55d37fe220ca] Ignoring malparsed backtrace: 2023-01-13T10:18:32.189 INFO:teuthology.orchestra.run.smithi077.stdout: , 2023-01-13T10:18:32.190 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":136,"fsid":"cb60268d-e37a-4a7a-85fb-ee1f96a8d263","created":"2023-01-13T09:47:58.104752+0000","modified":"2023-01-13T10:18:31.513310+0000","last_up_change":"2023-01-13T09:48:06.088475+0000","last_in_change":"2023-01-13T09:48:00.445778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-13T09:48:07.779681+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-13T09:48:08.702557+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-01-13T09:48:15.560751+0000","flags":262305,"flags_names":"hashpspool,nopgchange,write_fadvise_dontneed,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"b348d66e-ea99-4059-a761-8218d83a739b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":101993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6801","nonce":101993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":101993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6805","nonce":101993}]},"public_addr":"172.21.15.77:6800/101993","cluster_addr":"172.21.15.77:6801/101993","heartbeat_back_addr":"172.21.15.77:6804/101993","heartbeat_front_addr":"172.21.15.77:6805/101993","state":["exists","up"]},{"osd":1,"uuid":"80963b8f-6919-4dc0-8c42-cb45a8d27295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6803","nonce":102020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":102020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":102020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":102020}]},"public_addr":"172.21.15.77:6803/102020","cluster_addr":"172.21.15.77:6802/102020","heartbeat_back_addr":"172.21.15.77:6807/102020","heartbeat_front_addr":"172.21.15.77:6806/102020","state":["exists","up"]},{"osd":2,"uuid":"aa19a7b7-3a48-482e-93a6-5abe400f210c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":101672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":101672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":101672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":101672}]},"public_addr":"172.21.15.93:6803/101672","cluster_addr":"172.21.15.93:6802/101672","heartbeat_back_addr":"172.21.15.93:6806/101672","heartbeat_front_addr":"172.21.15.93:6807/101672","state":["exists","up"]},{"osd":3,"uuid":"0408e7fb-f03e-42c0-ad91-ce0a90e7d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":101700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":101700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6809","nonce":101700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":101700}]},"public_addr":"172.21.15.93:6805/101700","cluster_addr":"172.21.15.93:6804/101700","heartbeat_back_addr":"172.21.15.93:6809/101700","heartbeat_front_addr":"172.21.15.93:6808/101700","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2023-01-13T10:18:32.205 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats Ignoring malparsed backtrace: 2023-01-13T10:18:36.750 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-01-13T10:18:36.871+0000 7f82fee44700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-01-13T10:18:36.877+0000 7f82fee44700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-01-13T10:19:13.541 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/log/ceph -- . , 2023-01-13T10:19:17.961 DEBUG:teuthology.misc:Transferring archived files from smithi093:/var/log/ceph to /home/teuthworker/archive/sjust-2023-01-13_09:24:42-crimson-rados-wip-sjust-testing-2023-01-12-223717-distro-default-smithi/7141074/remote/smithi093/log , 2023-01-13T10:19:17.963 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar cz -f - -C /var/log/ceph -- . Ignoring malparsed backtrace: 2023-01-13T10:19:43.296 INFO:teuthology.task.internal: core looks like: /tmp/tmp14qlen2z: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-', real uid: 1000, effective uid: 1000, real gid: 1268, effective gid: 1268, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64' , , 2023-01-13T10:19:43.347 INFO:teuthology.task.internal: dump_program: ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in- Found 8 distinct failure reasons Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 1 jobs: ['7141070'] suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] Failure: reached maximum tries (500) after waiting for 3000 seconds 2 jobs: ['7141066', '7141069'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}'] Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_journal.sh 1 jobs: ['7141067'] suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_cls_tests}'] Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh 1 jobs: ['7141064'] suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}'] Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 1 jobs: ['7141073'] suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] Failure: Command failed on smithi006 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' 1 jobs: ['7141065'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2 jobs: ['7141071', '7141074'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}', 'workloads/write_fadvise_dontneed}'] Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 1 jobs: ['7141063'] suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}']