Found 367 jobs Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f509eb1da2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55da28bcad60] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-26T08:48:50.951 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-26T08:48:51.247 INFO:teuthology.orchestra.run.smithi013.stdout: Ignoring malparsed backtrace: 2020-12-26T08:50:13.350 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T08:50:16.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s , 2020-12-26T08:50:19.452 INFO:tasks.ceph.osd.5:Stopped Ignoring malparsed backtrace: 2020-12-26T08:52:10.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-12-26T08:52:10.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-12-26T08:52:10.635 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f22fb00e826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55c40baeffe6] Ignoring malparsed backtrace: 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4381e-05 , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2020-12-26T11:07:59.781651+0000", , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019e-05 , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, Ignoring malparsed backtrace: 2020-12-26T11:08:10.862+0000 7fbd63052700 -1 osd.7 68 handle_osd_map injecting map crc failure , 2020-12-26T11:08:10.987+0000 7f03bbbc8700 -1 mon.b@0(leader).osd e69 definitely_dead 0 , osd.2 is already down. Ignoring malparsed backtrace: 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", Ignoring malparsed backtrace: 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.315 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 13, , 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 37, , 2020-12-26T11:09:11.338 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 46, Ignoring malparsed backtrace: 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.41:6803/30289", , 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.41:6807/30289", , 2020-12-26T11:09:11.360 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.41:6805/30289", Ignoring malparsed backtrace: 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.41:6811", , 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 30319 , 2020-12-26T11:09:11.382 INFO:teuthology.orchestra.run.smithi097.stdout: } Ignoring malparsed backtrace: 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout: } , 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout:} , 2020-12-26T11:09:11.427 INFO:tasks.thrashosds.thrasher:j is {'epoch': 79, 'fsid': '46f57bee-5f7e-429d-81f8-d0943118d042', 'created': '2020-12-26T11:06:11.553513+0000', 'modified': '2020-12-26T11:09:08.655412+0000', 'last_up_change': '2020-12-26T11:09:07.645903+0000', 'last_in_change': '2020-12-26T11:08:11.729887+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-26T11:06:15.200377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-26T11:06:22.371765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 18, 'pg_placement_num': 18, 'pg_placement_num_target': 18, 'pg_num_target': 18, 'pg_num_pending': 18, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'base', 'create_time': '2020-12-26T11:06:35.626198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 4, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [4], 'tier_of': -1, 'read_tier': 4, 'write_tier': 4, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'myprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 4, 'pool_name': 'cache', 'create_time': '2020-12-26T11:06:38.963837+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 3, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 5000, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '30bc335d-671a-49f8-ad9a-b8e11ea38d51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 30311}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6801', 'nonce': 30311}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6803', 'nonce': 30311}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 30311}]}, 'public_addr': '172.21.15.97:6800/30311', 'cluster_addr': '172.21.15.97:6801/30311', 'heartbeat_back_addr': '172.21.15.97:6803/30311', 'heartbeat_front_addr': '172.21.15.97:6802/30311', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2728a643-5eb6-4f92-ba3e-9145f0784846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 30327}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 30327}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 30327}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 30327}]}, 'public_addr': '172.21.15.97:6808/30327', 'cluster_addr': '172.21.15.97:6809/30327', 'heartbeat_back_addr': '172.21.15.97:6811/30327', 'heartbeat_front_addr': '172.21.15.97:6810/30327', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1d8ad31b-031c-450b-8c81-fb313df9ecd7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 68, 'up_from': 78, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 52718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6813', 'nonce': 52718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6815', 'nonce': 52718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 52718}]}, 'public_addr': '172.21.15.97:6812/52718', 'cluster_addr': '172.21.15.97:6813/52718', 'heartbeat_back_addr': '172.21.15.97:6815/52718', 'heartbeat_front_addr': '172.21.15.97:6814/52718', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6c54189f-2275-477b-bcc0-5e45e66510bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 37, 'up_from': 46, 'up_thru': 70, 'down_at': 38, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 43460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 43460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6807', 'nonce': 43460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 43460}]}, 'public_addr': '172.21.15.97:6804/43460', 'cluster_addr': '172.21.15.97:6805/43460', 'heartbeat_back_addr': '172.21.15.97:6807/43460', 'heartbeat_front_addr': '172.21.15.97:6806/43460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fe41f56e-bc2e-44a3-bb66-6fce32511467', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6801', 'nonce': 30289}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6803', 'nonce': 30289}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6807', 'nonce': 30289}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6805', 'nonce': 30289}]}, 'public_addr': '172.21.15.41:6801/30289', 'cluster_addr': '172.21.15.41:6803/30289', 'heartbeat_back_addr': '172.21.15.41:6807/30289', 'heartbeat_front_addr': '172.21.15.41:6805/30289', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1d05d8b9-8cb2-444c-a590-7a941fa1e189', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6800', 'nonce': 30290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6802', 'nonce': 30290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6806', 'nonce': 30290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6804', 'nonce': 30290}]}, 'public_addr': '172.21.15.41:6800/30290', 'cluster_addr': '172.21.15.41:6802/30290', 'heartbeat_back_addr': '172.21.15.41:6806/30290', 'heartbeat_front_addr': '172.21.15.41:6804/30290', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b0a431da-859f-40ab-a41d-82216ca6c218', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6808', 'nonce': 30319}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6809', 'nonce': 30319}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6811', 'nonce': 30319}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6810', 'nonce': 30319}]}, 'public_addr': '172.21.15.41:6808/30319', 'cluster_addr': '172.21.15.41:6809/30319', 'heartbeat_back_addr': '172.21.15.41:6811/30319', 'heartbeat_front_addr': '172.21.15.41:6810/30319', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ce634247-eafd-40d6-9ffb-afb7893206f4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6812', 'nonce': 30320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6813', 'nonce': 30320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6815', 'nonce': 30320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6814', 'nonce': 30320}]}, 'public_addr': '172.21.15.41:6812/30320', 'cluster_addr': '172.21.15.41:6813/30320', 'heartbeat_back_addr': '172.21.15.41:6815/30320', 'heartbeat_front_addr': '172.21.15.41:6814/30320', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.314520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.867141+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:09:08.648413+0000', 'dead_epoch': 69}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:20.277598+0000', 'dead_epoch': 38}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:17.969211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.121401+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.174114+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.323352+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'myprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '2', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} Ignoring malparsed backtrace: 2020-12-26T11:09:37.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats , 2020-12-26T11:09:37.691 INFO:teuthology.orchestra.run.smithi097.stdout:55834574890 , 2020-12-26T11:09:37.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats Ignoring malparsed backtrace: 2020-12-26T11:09:46.888 INFO:teuthology.orchestra.run.smithi097.stdout:{ , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout:} Ignoring malparsed backtrace: 2020-12-26T11:10:29.194 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-12-26T11:10:29.195 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T11:10:29.261 INFO:tasks.ceph.osd.1:Stopped Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested , yield vars Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get , File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f22fb00e826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55c40baeffe6] Ignoring malparsed backtrace: 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4381e-05 , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2020-12-26T11:07:59.781651+0000", , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019e-05 , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, Ignoring malparsed backtrace: 2020-12-26T11:08:10.862+0000 7fbd63052700 -1 osd.7 68 handle_osd_map injecting map crc failure , 2020-12-26T11:08:10.987+0000 7f03bbbc8700 -1 mon.b@0(leader).osd e69 definitely_dead 0 , osd.2 is already down. Ignoring malparsed backtrace: 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", Ignoring malparsed backtrace: 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.315 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 13, , 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 37, , 2020-12-26T11:09:11.338 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 46, Ignoring malparsed backtrace: 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.41:6803/30289", , 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.41:6807/30289", , 2020-12-26T11:09:11.360 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.41:6805/30289", Ignoring malparsed backtrace: 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.41:6811", , 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 30319 , 2020-12-26T11:09:11.382 INFO:teuthology.orchestra.run.smithi097.stdout: } Ignoring malparsed backtrace: 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout: } , 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout:} , 2020-12-26T11:09:11.427 INFO:tasks.thrashosds.thrasher:j is {'epoch': 79, 'fsid': '46f57bee-5f7e-429d-81f8-d0943118d042', 'created': '2020-12-26T11:06:11.553513+0000', 'modified': '2020-12-26T11:09:08.655412+0000', 'last_up_change': '2020-12-26T11:09:07.645903+0000', 'last_in_change': '2020-12-26T11:08:11.729887+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-26T11:06:15.200377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-26T11:06:22.371765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 18, 'pg_placement_num': 18, 'pg_placement_num_target': 18, 'pg_num_target': 18, 'pg_num_pending': 18, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'base', 'create_time': '2020-12-26T11:06:35.626198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 4, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [4], 'tier_of': -1, 'read_tier': 4, 'write_tier': 4, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'myprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 4, 'pool_name': 'cache', 'create_time': '2020-12-26T11:06:38.963837+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 3, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 5000, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '30bc335d-671a-49f8-ad9a-b8e11ea38d51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 30311}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6801', 'nonce': 30311}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6803', 'nonce': 30311}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 30311}]}, 'public_addr': '172.21.15.97:6800/30311', 'cluster_addr': '172.21.15.97:6801/30311', 'heartbeat_back_addr': '172.21.15.97:6803/30311', 'heartbeat_front_addr': '172.21.15.97:6802/30311', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2728a643-5eb6-4f92-ba3e-9145f0784846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 30327}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 30327}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 30327}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 30327}]}, 'public_addr': '172.21.15.97:6808/30327', 'cluster_addr': '172.21.15.97:6809/30327', 'heartbeat_back_addr': '172.21.15.97:6811/30327', 'heartbeat_front_addr': '172.21.15.97:6810/30327', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1d8ad31b-031c-450b-8c81-fb313df9ecd7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 68, 'up_from': 78, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 52718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6813', 'nonce': 52718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6815', 'nonce': 52718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 52718}]}, 'public_addr': '172.21.15.97:6812/52718', 'cluster_addr': '172.21.15.97:6813/52718', 'heartbeat_back_addr': '172.21.15.97:6815/52718', 'heartbeat_front_addr': '172.21.15.97:6814/52718', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6c54189f-2275-477b-bcc0-5e45e66510bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 37, 'up_from': 46, 'up_thru': 70, 'down_at': 38, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 43460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 43460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6807', 'nonce': 43460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 43460}]}, 'public_addr': '172.21.15.97:6804/43460', 'cluster_addr': '172.21.15.97:6805/43460', 'heartbeat_back_addr': '172.21.15.97:6807/43460', 'heartbeat_front_addr': '172.21.15.97:6806/43460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fe41f56e-bc2e-44a3-bb66-6fce32511467', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6801', 'nonce': 30289}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6803', 'nonce': 30289}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6807', 'nonce': 30289}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6805', 'nonce': 30289}]}, 'public_addr': '172.21.15.41:6801/30289', 'cluster_addr': '172.21.15.41:6803/30289', 'heartbeat_back_addr': '172.21.15.41:6807/30289', 'heartbeat_front_addr': '172.21.15.41:6805/30289', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1d05d8b9-8cb2-444c-a590-7a941fa1e189', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6800', 'nonce': 30290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6802', 'nonce': 30290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6806', 'nonce': 30290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6804', 'nonce': 30290}]}, 'public_addr': '172.21.15.41:6800/30290', 'cluster_addr': '172.21.15.41:6802/30290', 'heartbeat_back_addr': '172.21.15.41:6806/30290', 'heartbeat_front_addr': '172.21.15.41:6804/30290', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b0a431da-859f-40ab-a41d-82216ca6c218', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6808', 'nonce': 30319}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6809', 'nonce': 30319}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6811', 'nonce': 30319}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6810', 'nonce': 30319}]}, 'public_addr': '172.21.15.41:6808/30319', 'cluster_addr': '172.21.15.41:6809/30319', 'heartbeat_back_addr': '172.21.15.41:6811/30319', 'heartbeat_front_addr': '172.21.15.41:6810/30319', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ce634247-eafd-40d6-9ffb-afb7893206f4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6812', 'nonce': 30320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6813', 'nonce': 30320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6815', 'nonce': 30320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6814', 'nonce': 30320}]}, 'public_addr': '172.21.15.41:6812/30320', 'cluster_addr': '172.21.15.41:6813/30320', 'heartbeat_back_addr': '172.21.15.41:6815/30320', 'heartbeat_front_addr': '172.21.15.41:6814/30320', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.314520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.867141+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:09:08.648413+0000', 'dead_epoch': 69}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:20.277598+0000', 'dead_epoch': 38}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:17.969211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.121401+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.174114+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.323352+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'myprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '2', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} Ignoring malparsed backtrace: 2020-12-26T11:09:37.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats , 2020-12-26T11:09:37.691 INFO:teuthology.orchestra.run.smithi097.stdout:55834574890 , 2020-12-26T11:09:37.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats Ignoring malparsed backtrace: 2020-12-26T11:09:46.888 INFO:teuthology.orchestra.run.smithi097.stdout:{ , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout:} Ignoring malparsed backtrace: 2020-12-26T11:10:29.194 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-12-26T11:10:29.195 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T11:10:29.261 INFO:tasks.ceph.osd.1:Stopped Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested , yield vars Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get , File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f22fb00e826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55c40baeffe6] Ignoring malparsed backtrace: 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4381e-05 , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2020-12-26T11:07:59.781651+0000", , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019e-05 , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, Ignoring malparsed backtrace: 2020-12-26T11:08:10.862+0000 7fbd63052700 -1 osd.7 68 handle_osd_map injecting map crc failure , 2020-12-26T11:08:10.987+0000 7f03bbbc8700 -1 mon.b@0(leader).osd e69 definitely_dead 0 , osd.2 is already down. Ignoring malparsed backtrace: 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", Ignoring malparsed backtrace: 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.315 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 13, , 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 37, , 2020-12-26T11:09:11.338 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 46, Ignoring malparsed backtrace: 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.41:6803/30289", , 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.41:6807/30289", , 2020-12-26T11:09:11.360 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.41:6805/30289", Ignoring malparsed backtrace: 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.41:6811", , 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 30319 , 2020-12-26T11:09:11.382 INFO:teuthology.orchestra.run.smithi097.stdout: } Ignoring malparsed backtrace: 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout: } , 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout:} , 2020-12-26T11:09:11.427 INFO:tasks.thrashosds.thrasher:j is {'epoch': 79, 'fsid': '46f57bee-5f7e-429d-81f8-d0943118d042', 'created': '2020-12-26T11:06:11.553513+0000', 'modified': '2020-12-26T11:09:08.655412+0000', 'last_up_change': '2020-12-26T11:09:07.645903+0000', 'last_in_change': '2020-12-26T11:08:11.729887+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-26T11:06:15.200377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-26T11:06:22.371765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 18, 'pg_placement_num': 18, 'pg_placement_num_target': 18, 'pg_num_target': 18, 'pg_num_pending': 18, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'base', 'create_time': '2020-12-26T11:06:35.626198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 4, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [4], 'tier_of': -1, 'read_tier': 4, 'write_tier': 4, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'myprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 4, 'pool_name': 'cache', 'create_time': '2020-12-26T11:06:38.963837+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 3, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 5000, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '30bc335d-671a-49f8-ad9a-b8e11ea38d51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 30311}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6801', 'nonce': 30311}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6803', 'nonce': 30311}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 30311}]}, 'public_addr': '172.21.15.97:6800/30311', 'cluster_addr': '172.21.15.97:6801/30311', 'heartbeat_back_addr': '172.21.15.97:6803/30311', 'heartbeat_front_addr': '172.21.15.97:6802/30311', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2728a643-5eb6-4f92-ba3e-9145f0784846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 30327}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 30327}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 30327}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 30327}]}, 'public_addr': '172.21.15.97:6808/30327', 'cluster_addr': '172.21.15.97:6809/30327', 'heartbeat_back_addr': '172.21.15.97:6811/30327', 'heartbeat_front_addr': '172.21.15.97:6810/30327', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1d8ad31b-031c-450b-8c81-fb313df9ecd7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 68, 'up_from': 78, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 52718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6813', 'nonce': 52718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6815', 'nonce': 52718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 52718}]}, 'public_addr': '172.21.15.97:6812/52718', 'cluster_addr': '172.21.15.97:6813/52718', 'heartbeat_back_addr': '172.21.15.97:6815/52718', 'heartbeat_front_addr': '172.21.15.97:6814/52718', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6c54189f-2275-477b-bcc0-5e45e66510bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 37, 'up_from': 46, 'up_thru': 70, 'down_at': 38, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 43460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 43460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6807', 'nonce': 43460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 43460}]}, 'public_addr': '172.21.15.97:6804/43460', 'cluster_addr': '172.21.15.97:6805/43460', 'heartbeat_back_addr': '172.21.15.97:6807/43460', 'heartbeat_front_addr': '172.21.15.97:6806/43460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fe41f56e-bc2e-44a3-bb66-6fce32511467', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6801', 'nonce': 30289}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6803', 'nonce': 30289}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6807', 'nonce': 30289}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6805', 'nonce': 30289}]}, 'public_addr': '172.21.15.41:6801/30289', 'cluster_addr': '172.21.15.41:6803/30289', 'heartbeat_back_addr': '172.21.15.41:6807/30289', 'heartbeat_front_addr': '172.21.15.41:6805/30289', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1d05d8b9-8cb2-444c-a590-7a941fa1e189', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6800', 'nonce': 30290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6802', 'nonce': 30290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6806', 'nonce': 30290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6804', 'nonce': 30290}]}, 'public_addr': '172.21.15.41:6800/30290', 'cluster_addr': '172.21.15.41:6802/30290', 'heartbeat_back_addr': '172.21.15.41:6806/30290', 'heartbeat_front_addr': '172.21.15.41:6804/30290', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b0a431da-859f-40ab-a41d-82216ca6c218', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6808', 'nonce': 30319}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6809', 'nonce': 30319}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6811', 'nonce': 30319}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6810', 'nonce': 30319}]}, 'public_addr': '172.21.15.41:6808/30319', 'cluster_addr': '172.21.15.41:6809/30319', 'heartbeat_back_addr': '172.21.15.41:6811/30319', 'heartbeat_front_addr': '172.21.15.41:6810/30319', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ce634247-eafd-40d6-9ffb-afb7893206f4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6812', 'nonce': 30320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6813', 'nonce': 30320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6815', 'nonce': 30320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6814', 'nonce': 30320}]}, 'public_addr': '172.21.15.41:6812/30320', 'cluster_addr': '172.21.15.41:6813/30320', 'heartbeat_back_addr': '172.21.15.41:6815/30320', 'heartbeat_front_addr': '172.21.15.41:6814/30320', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.314520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.867141+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:09:08.648413+0000', 'dead_epoch': 69}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:20.277598+0000', 'dead_epoch': 38}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:17.969211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.121401+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.174114+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.323352+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'myprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '2', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} Ignoring malparsed backtrace: 2020-12-26T11:09:37.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats , 2020-12-26T11:09:37.691 INFO:teuthology.orchestra.run.smithi097.stdout:55834574890 , 2020-12-26T11:09:37.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats Ignoring malparsed backtrace: 2020-12-26T11:09:46.888 INFO:teuthology.orchestra.run.smithi097.stdout:{ , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout:} Ignoring malparsed backtrace: 2020-12-26T11:10:29.194 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-12-26T11:10:29.195 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T11:10:29.261 INFO:tasks.ceph.osd.1:Stopped Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested , yield vars Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get , File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception Ignoring malparsed backtrace: ceph version 16.0.0-8584-g43e363d3 (43e363d3957ee0c8e0e948d4d3e58a75abda2778) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f22fb00e826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55c40baeffe6] Ignoring malparsed backtrace: 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4381e-05 , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:07:59.803 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2020-12-26T11:07:59.781651+0000", , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019e-05 , 2020-12-26T11:07:59.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, Ignoring malparsed backtrace: 2020-12-26T11:08:10.862+0000 7fbd63052700 -1 osd.7 68 handle_osd_map injecting map crc failure , 2020-12-26T11:08:10.987+0000 7f03bbbc8700 -1 mon.b@0(leader).osd e69 definitely_dead 0 , osd.2 is already down. Ignoring malparsed backtrace: 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "pglog_hardlimit", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "purged_snapdirs", , 2020-12-26T11:09:11.225 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_deletes", Ignoring malparsed backtrace: 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_count": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_target": 0, , 2020-12-26T11:09:11.248 INFO:teuthology.orchestra.run.smithi097.stdout: "peering_crush_bucket_barrier": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "removed_snaps": "[]", , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_bytes": 0, , 2020-12-26T11:09:11.271 INFO:teuthology.orchestra.run.smithi097.stdout: "quota_max_objects": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "min_write_recency_for_promote": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_grade_decay_rate": 0, , 2020-12-26T11:09:11.293 INFO:teuthology.orchestra.run.smithi097.stdout: "hit_set_search_last_n": 0, Ignoring malparsed backtrace: 2020-12-26T11:09:11.315 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addrs": { , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ , 2020-12-26T11:09:11.316 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_begin": 13, , 2020-12-26T11:09:11.337 INFO:teuthology.orchestra.run.smithi097.stdout: "last_clean_end": 37, , 2020-12-26T11:09:11.338 INFO:teuthology.orchestra.run.smithi097.stdout: "up_from": 46, Ignoring malparsed backtrace: 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_addr": "172.21.15.41:6803/30289", , 2020-12-26T11:09:11.359 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_back_addr": "172.21.15.41:6807/30289", , 2020-12-26T11:09:11.360 INFO:teuthology.orchestra.run.smithi097.stdout: "heartbeat_front_addr": "172.21.15.41:6805/30289", Ignoring malparsed backtrace: 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.41:6811", , 2020-12-26T11:09:11.381 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 30319 , 2020-12-26T11:09:11.382 INFO:teuthology.orchestra.run.smithi097.stdout: } Ignoring malparsed backtrace: 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: "dead_epoch": 0 , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, , 2020-12-26T11:09:11.403 INFO:teuthology.orchestra.run.smithi097.stdout: { Ignoring malparsed backtrace: 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout: } , 2020-12-26T11:09:11.425 INFO:teuthology.orchestra.run.smithi097.stdout:} , 2020-12-26T11:09:11.427 INFO:tasks.thrashosds.thrasher:j is {'epoch': 79, 'fsid': '46f57bee-5f7e-429d-81f8-d0943118d042', 'created': '2020-12-26T11:06:11.553513+0000', 'modified': '2020-12-26T11:09:08.655412+0000', 'last_up_change': '2020-12-26T11:09:07.645903+0000', 'last_in_change': '2020-12-26T11:08:11.729887+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-26T11:06:15.200377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-26T11:06:22.371765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 18, 'pg_placement_num': 18, 'pg_placement_num_target': 18, 'pg_num_target': 18, 'pg_num_pending': 18, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'base', 'create_time': '2020-12-26T11:06:35.626198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 4, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [4], 'tier_of': -1, 'read_tier': 4, 'write_tier': 4, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'myprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 4, 'pool_name': 'cache', 'create_time': '2020-12-26T11:06:38.963837+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '28', 'last_force_op_resend_prenautilus': '28', 'last_force_op_resend_preluminous': '28', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 3, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 5000, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '30bc335d-671a-49f8-ad9a-b8e11ea38d51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6800', 'nonce': 30311}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6801', 'nonce': 30311}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6803', 'nonce': 30311}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6802', 'nonce': 30311}]}, 'public_addr': '172.21.15.97:6800/30311', 'cluster_addr': '172.21.15.97:6801/30311', 'heartbeat_back_addr': '172.21.15.97:6803/30311', 'heartbeat_front_addr': '172.21.15.97:6802/30311', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2728a643-5eb6-4f92-ba3e-9145f0784846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6808', 'nonce': 30327}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6809', 'nonce': 30327}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6811', 'nonce': 30327}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6810', 'nonce': 30327}]}, 'public_addr': '172.21.15.97:6808/30327', 'cluster_addr': '172.21.15.97:6809/30327', 'heartbeat_back_addr': '172.21.15.97:6811/30327', 'heartbeat_front_addr': '172.21.15.97:6810/30327', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1d8ad31b-031c-450b-8c81-fb313df9ecd7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 68, 'up_from': 78, 'up_thru': 46, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6812', 'nonce': 52718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6813', 'nonce': 52718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6815', 'nonce': 52718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6814', 'nonce': 52718}]}, 'public_addr': '172.21.15.97:6812/52718', 'cluster_addr': '172.21.15.97:6813/52718', 'heartbeat_back_addr': '172.21.15.97:6815/52718', 'heartbeat_front_addr': '172.21.15.97:6814/52718', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6c54189f-2275-477b-bcc0-5e45e66510bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 37, 'up_from': 46, 'up_thru': 70, 'down_at': 38, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6804', 'nonce': 43460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6805', 'nonce': 43460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6807', 'nonce': 43460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6806', 'nonce': 43460}]}, 'public_addr': '172.21.15.97:6804/43460', 'cluster_addr': '172.21.15.97:6805/43460', 'heartbeat_back_addr': '172.21.15.97:6807/43460', 'heartbeat_front_addr': '172.21.15.97:6806/43460', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fe41f56e-bc2e-44a3-bb66-6fce32511467', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6801', 'nonce': 30289}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6803', 'nonce': 30289}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6807', 'nonce': 30289}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6805', 'nonce': 30289}]}, 'public_addr': '172.21.15.41:6801/30289', 'cluster_addr': '172.21.15.41:6803/30289', 'heartbeat_back_addr': '172.21.15.41:6807/30289', 'heartbeat_front_addr': '172.21.15.41:6805/30289', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1d05d8b9-8cb2-444c-a590-7a941fa1e189', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6800', 'nonce': 30290}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6802', 'nonce': 30290}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6806', 'nonce': 30290}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6804', 'nonce': 30290}]}, 'public_addr': '172.21.15.41:6800/30290', 'cluster_addr': '172.21.15.41:6802/30290', 'heartbeat_back_addr': '172.21.15.41:6806/30290', 'heartbeat_front_addr': '172.21.15.41:6804/30290', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b0a431da-859f-40ab-a41d-82216ca6c218', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6808', 'nonce': 30319}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6809', 'nonce': 30319}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6811', 'nonce': 30319}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6810', 'nonce': 30319}]}, 'public_addr': '172.21.15.41:6808/30319', 'cluster_addr': '172.21.15.41:6809/30319', 'heartbeat_back_addr': '172.21.15.41:6811/30319', 'heartbeat_front_addr': '172.21.15.41:6810/30319', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ce634247-eafd-40d6-9ffb-afb7893206f4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6812', 'nonce': 30320}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6813', 'nonce': 30320}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6815', 'nonce': 30320}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.41:6814', 'nonce': 30320}]}, 'public_addr': '172.21.15.41:6812/30320', 'cluster_addr': '172.21.15.41:6813/30320', 'heartbeat_back_addr': '172.21.15.41:6815/30320', 'heartbeat_front_addr': '172.21.15.41:6814/30320', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.314520+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.867141+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:09:08.648413+0000', 'dead_epoch': 69}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:20.277598+0000', 'dead_epoch': 38}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:17.969211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.121401+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.174114+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2020-12-26T11:08:18.323352+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'myprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '2', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} Ignoring malparsed backtrace: 2020-12-26T11:09:37.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats , 2020-12-26T11:09:37.691 INFO:teuthology.orchestra.run.smithi097.stdout:55834574890 , 2020-12-26T11:09:37.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats Ignoring malparsed backtrace: 2020-12-26T11:09:46.888 INFO:teuthology.orchestra.run.smithi097.stdout:{ , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-26T11:09:46.889 INFO:teuthology.orchestra.run.smithi097.stdout:} Ignoring malparsed backtrace: 2020-12-26T11:10:29.194 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-12-26T11:10:29.195 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-26T11:10:29.261 INFO:tasks.ceph.osd.1:Stopped Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested , yield vars Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get , File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception Found 9 distinct failure reasons Failure: reached maximum tries (180) after waiting for 180 seconds 4 jobs: ['5737972', '5737878', '5737941', '5738104'] suites intersection: [] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'distro$/{centos_8.0}', 'distro$/{centos_latest}', 'distro$/{rhel_8.0}', 'fixed-2', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity}', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/cephadm/upgrade/{1-start', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] Failure: "2020-12-26T09:57:15.727976+0000 osd.0 (osd.0) 143 : cluster [ERR] scrub 59.1d 59:bbaa2602:test-rados-api-smithi007-85713-74::foo:24 : size 0 != clone_size 10" in cluster log 1 jobs: ['5737927'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] Failure: "2020-12-26T10:22:23.521304+0000 mon.a (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log 1 jobs: ['5738019'] suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/osd-recovery-incomplete', 'supported-random-distro$/{rhel_8}}'] Dead: b'2020-12-26T22:15:25.573 DEBUG:teuthology.exit:Finished running handlers' 9 jobs: ['5738030', '5738021', '5738004', '5737963', '5737939', '5737871', '5737757', '5737752', '5737970'] suites intersection: ['openstack}', 'rados', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-pool-snaps}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-small-objects-many-deletes}', 'workloads/redirect}', 'workloads/small-objects-localized}', 'workloads/small-objects}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2 jobs: ['5737826', '5738100'] suites intersection: ['mon_election/classic', 'msgr-failures/few', 'rados'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-agent-big}'] Failure: Command failed (workunit test cephtool/test.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43e363d3957ee0c8e0e948d4d3e58a75abda2778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 1 jobs: ['5737767'] suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_8}}'] Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2 jobs: ['5737901', '5738044'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] Timeout 120 running ceph --cluster ceph osd dump --format=json 1 jobs: ['5737756'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/small-objects-localized}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43e363d3957ee0c8e0e948d4d3e58a75abda2778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2 jobs: ['5737953', '5737819'] suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']