Found 194 jobs Ignoring malparsed backtrace: ceph version 16.0.0-4450-gbd745e79 (bd745e79ac867292f9946dc3bbde9d10caea3ea4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f8abd352bd4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x555c79d4d686] Ignoring malparsed backtrace: 2020-12-18T07:51:42.225 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops , 2020-12-18T07:51:42.347 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument , 2020-12-18T07:51:42.348 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-18T07:52:05.553 INFO:teuthology.orchestra.run.smithi055.stdout: , 2020-12-18T07:52:05.553 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":292,"fsid":"0b10911c-55f2-4321-98ee-e48f824cb951","created":"2020-12-18T07:45:58.166622+0000","modified":"2020-12-18T07:52:04.971452+0000","last_up_change":"2020-12-18T07:51:37.993562+0000","last_in_change":"2020-12-18T07:51:48.956269+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-18T07:46:01.658540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":256,"last_epoch_started":255,"last_epoch_clean":255,"source_version":"15'2","target_version":"15'2"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"256","last_force_op_resend_preluminous":"254","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-12-18T07:46:12.073110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-12-18T07:46:26.099909+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":124,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"c52faba0-9abc-44de-aa9b-6c251284ece1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":58,"up_from":214,"up_thru":290,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":67104},{"type":"v1","addr":"172.21.15.55:6803","nonce":67104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":67104},{"type":"v1","addr":"172.21.15.55:6805","nonce":67104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":67104},{"type":"v1","addr":"172.21.15.55:6809","nonce":67104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":67104},{"type":"v1","addr":"172.21.15.55:6807","nonce":67104}]},"public_addr":"172.21.15.55:6803/67104","cluster_addr":"172.21.15.55:6805/67104","heartbeat_back_addr":"172.21.15.55:6809/67104","heartbeat_front_addr":"172.21.15.55:6807/67104","state":["exists","up"]},{"osd":1,"uuid":"4fcb76ba-fd13-47b6-b4b5-209850e4660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":108,"up_from":221,"up_thru":291,"down_at":109,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":68578},{"type":"v1","addr":"172.21.15.55:6811","nonce":68578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":68578},{"type":"v1","addr":"172.21.15.55:6813","nonce":68578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":68578},{"type":"v1","addr":"172.21.15.55:6817","nonce":68578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":68578},{"type":"v1","addr":"172.21.15.55:6815","nonce":68578}]},"public_addr":"172.21.15.55:6811/68578","cluster_addr":"172.21.15.55:6813/68578","heartbeat_back_addr":"172.21.15.55:6817/68578","heartbeat_front_addr":"172.21.15.55:6815/68578","state":["exists","up"]},{"osd":2,"uuid":"2aabf52a-73a6-403d-b764-a86024eea78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":31473},{"type":"v1","addr":"172.21.15.55:6819","nonce":31473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":31473},{"type":"v1","addr":"172.21.15.55:6822","nonce":31473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6827","nonce":31473},{"type":"v1","addr":"172.21.15.55:6829","nonce":31473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6823","nonce":31473},{"type":"v1","addr":"172.21.15.55:6825","nonce":31473}]},"public_addr":"172.21.15.55:6819/31473","cluster_addr":"172.21.15.55:6822/31473","heartbeat_back_addr":"172.21.15.55:6829/31473","heartbeat_front_addr":"172.21.15.55:6825/31473","state":["exists","up"]},{"osd":3,"uuid":"5f2a1b95-ed63-4e0a-bb62-c935eb2ad4ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6821","nonce":31474},{"type":"v1","addr":"172.21.15.55:6824","nonce":31474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":31474},{"type":"v1","addr":"172.21.15.55:6828","nonce":31474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":31474},{"type":"v1","addr":"172.21.15.55:6833","nonce":31474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":31474},{"type":"v1","addr":"172.21.15.55:6831","nonce":31474}]},"public_addr":"172.21.15.55:6824/31474","cluster_addr":"172.21.15.55:6828/31474","heartbeat_back_addr":"172.21.15.55:6833/31474","heartbeat_front_addr":"172.21.15.55:6831/31474","state":["exists","up"]},{"osd":4,"uuid":"271d0b98-4bad-4157-bd46-d301aeeaf7d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":176,"up_from":199,"up_thru":281,"down_at":177,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":54744},{"type":"v1","addr":"172.21.15.125:6804","nonce":54744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":54744},{"type":"v1","addr":"172.21.15.125:6808","nonce":54744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":54744},{"type":"v1","addr":"172.21.15.125:6815","nonce":54744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":54744},{"type":"v1","addr":"172.21.15.125:6812","nonce":54744}]},"public_addr":"172.21.15.125:6804/54744","cluster_addr":"172.21.15.125:6808/54744","heartbeat_back_addr":"172.21.15.125:6815/54744","heartbeat_front_addr":"172.21.15.125:6812/54744","state":["exists","up"]},{"osd":5,"uuid":"a8ae81ab-36f1-4c1b-908b-4049b805a2b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":31378},{"type":"v1","addr":"172.21.15.125:6801","nonce":31378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6803","nonce":31378},{"type":"v1","addr":"172.21.15.125:6805","nonce":31378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":31378},{"type":"v1","addr":"172.21.15.125:6813","nonce":31378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6807","nonce":31378},{"type":"v1","addr":"172.21.15.125:6809","nonce":31378}]},"public_addr":"172.21.15.125:6801/31378","cluster_addr":"172.21.15.125:6805/31378","heartbeat_back_addr":"172.21.15.125:6813/31378","heartbeat_front_addr":"172.21.15.125:6809/31378","state":["exists","up"]},{"osd":6,"uuid":"e6d12493-c134-4cd9-88b5-a1e5eb2ad553","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":31379},{"type":"v1","addr":"172.21.15.125:6817","nonce":31379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":31379},{"type":"v1","addr":"172.21.15.125:6819","nonce":31379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":31379},{"type":"v1","addr":"172.21.15.125:6826","nonce":31379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":31379},{"type":"v1","addr":"172.21.15.125:6822","nonce":31379}]},"public_addr":"172.21.15.125:6817/31379","cluster_addr":"172.21.15.125:6819/31379","heartbeat_back_addr":"172.21.15.125:6826/31379","heartbeat_front_addr":"172.21.15.125:6822/31379","state":["exists","up"]},{"osd":7,"uuid":"3c4fce39-796b-492e-b19e-db661997ae80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":145,"up_from":265,"up_thru":278,"down_at":147,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":59463},{"type":"v1","addr":"172.21.15.125:6823","nonce":59463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":59463},{"type":"v1","addr":"172.21.15.125:6827","nonce":59463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":59463},{"type":"v1","addr":"172.21.15.125:6831","nonce":59463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":59463},{"type":"v1","addr":"172.21.15.125:6829","nonce":59463}]},"public_addr":"172.21.15.125:6823/59463","cluster_addr":"172.21.15.125:6827/59463","heartbeat_back_addr":"172.21.15.125:6831/59463","heartbeat_front_addr":"172.21.15.125:6829/59463","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:31.943518+0000","dead_epoch":59},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:42.554598+0000","dead_epoch":109},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:08.792704+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:09.199838+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:15.817034+0000","dead_epoch":177},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:07.780253+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:09.525009+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:38.275071+0000","dead_epoch":147}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.2","osds":[1,3,5]},{"pgid":"1.3","osds":[1,7,5]},{"pgid":"1.4","osds":[1,7,5]},{"pgid":"1.7","osds":[1,0,7]},{"pgid":"1.b","osds":[5,1,2]},{"pgid":"1.c","osds":[5,1,0]},{"pgid":"1.e","osds":[3,1,2]},{"pgid":"1.f","osds":[5,1,2]},{"pgid":"1.11","osds":[3,5,4]},{"pgid":"1.12","osds":[3,5,4]},{"pgid":"1.13","osds":[5,2,4]},{"pgid":"1.14","osds":[5,2,4]}],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} , 2020-12-18T07:52:05.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.0 flush_pg_stats Ignoring malparsed backtrace: 2020-12-18T07:52:58.334 INFO:teuthology.orchestra.run.smithi055.stdout:{ , 2020-12-18T07:52:58.334 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-18T07:52:58.334 INFO:teuthology.orchestra.run.smithi055.stdout:} Ignoring malparsed backtrace: 2020-12-18T07:53:22.125 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-12-18T07:53:34.399+0000 7f33d4590700 -1 --2- 172.21.15.55:0/4031903027 >> [v2:172.21.15.55:6803/31472,v1:172.21.15.55:6805/31472] conn(0x55c3774d2400 0x55c3776c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.55:6803/31472,v1:172.21.15.55:6805/31472] is using msgr V1 protocol , 2020-12-18T07:53:42.127 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json Ignoring malparsed backtrace: 2020-12-18T07:53:45.877 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.aQM5x5ZsNZ , 2020-12-18T07:53:45.892 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/shuzhenyi-2020-12-18_06:57:08-rados:thrash-erasure-code-wip-zhenyi-testing-2020-12-17-1108-distro-basic-smithi/5717770/data/mon.c.tgz , 2020-12-18T07:53:45.893 DEBUG:teuthology.orchestra.run.smithi055:> mktemp Ignoring malparsed backtrace: 2020-12-18T07:54:52.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service , Redirecting to /bin/systemctl restart rsyslog.service Ignoring malparsed backtrace: ceph version 16.0.0-4450-gbd745e79 (bd745e79ac867292f9946dc3bbde9d10caea3ea4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f8abd352bd4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x555c79d4d686] Ignoring malparsed backtrace: 2020-12-18T07:51:42.225 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops , 2020-12-18T07:51:42.347 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument , 2020-12-18T07:51:42.348 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-18T07:52:05.553 INFO:teuthology.orchestra.run.smithi055.stdout: , 2020-12-18T07:52:05.553 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":292,"fsid":"0b10911c-55f2-4321-98ee-e48f824cb951","created":"2020-12-18T07:45:58.166622+0000","modified":"2020-12-18T07:52:04.971452+0000","last_up_change":"2020-12-18T07:51:37.993562+0000","last_in_change":"2020-12-18T07:51:48.956269+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-18T07:46:01.658540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":256,"last_epoch_started":255,"last_epoch_clean":255,"source_version":"15'2","target_version":"15'2"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"256","last_force_op_resend_preluminous":"254","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-12-18T07:46:12.073110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-12-18T07:46:26.099909+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":124,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"c52faba0-9abc-44de-aa9b-6c251284ece1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":58,"up_from":214,"up_thru":290,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":67104},{"type":"v1","addr":"172.21.15.55:6803","nonce":67104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":67104},{"type":"v1","addr":"172.21.15.55:6805","nonce":67104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":67104},{"type":"v1","addr":"172.21.15.55:6809","nonce":67104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":67104},{"type":"v1","addr":"172.21.15.55:6807","nonce":67104}]},"public_addr":"172.21.15.55:6803/67104","cluster_addr":"172.21.15.55:6805/67104","heartbeat_back_addr":"172.21.15.55:6809/67104","heartbeat_front_addr":"172.21.15.55:6807/67104","state":["exists","up"]},{"osd":1,"uuid":"4fcb76ba-fd13-47b6-b4b5-209850e4660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":108,"up_from":221,"up_thru":291,"down_at":109,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":68578},{"type":"v1","addr":"172.21.15.55:6811","nonce":68578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":68578},{"type":"v1","addr":"172.21.15.55:6813","nonce":68578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":68578},{"type":"v1","addr":"172.21.15.55:6817","nonce":68578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":68578},{"type":"v1","addr":"172.21.15.55:6815","nonce":68578}]},"public_addr":"172.21.15.55:6811/68578","cluster_addr":"172.21.15.55:6813/68578","heartbeat_back_addr":"172.21.15.55:6817/68578","heartbeat_front_addr":"172.21.15.55:6815/68578","state":["exists","up"]},{"osd":2,"uuid":"2aabf52a-73a6-403d-b764-a86024eea78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":31473},{"type":"v1","addr":"172.21.15.55:6819","nonce":31473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":31473},{"type":"v1","addr":"172.21.15.55:6822","nonce":31473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6827","nonce":31473},{"type":"v1","addr":"172.21.15.55:6829","nonce":31473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6823","nonce":31473},{"type":"v1","addr":"172.21.15.55:6825","nonce":31473}]},"public_addr":"172.21.15.55:6819/31473","cluster_addr":"172.21.15.55:6822/31473","heartbeat_back_addr":"172.21.15.55:6829/31473","heartbeat_front_addr":"172.21.15.55:6825/31473","state":["exists","up"]},{"osd":3,"uuid":"5f2a1b95-ed63-4e0a-bb62-c935eb2ad4ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6821","nonce":31474},{"type":"v1","addr":"172.21.15.55:6824","nonce":31474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":31474},{"type":"v1","addr":"172.21.15.55:6828","nonce":31474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":31474},{"type":"v1","addr":"172.21.15.55:6833","nonce":31474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":31474},{"type":"v1","addr":"172.21.15.55:6831","nonce":31474}]},"public_addr":"172.21.15.55:6824/31474","cluster_addr":"172.21.15.55:6828/31474","heartbeat_back_addr":"172.21.15.55:6833/31474","heartbeat_front_addr":"172.21.15.55:6831/31474","state":["exists","up"]},{"osd":4,"uuid":"271d0b98-4bad-4157-bd46-d301aeeaf7d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":176,"up_from":199,"up_thru":281,"down_at":177,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":54744},{"type":"v1","addr":"172.21.15.125:6804","nonce":54744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":54744},{"type":"v1","addr":"172.21.15.125:6808","nonce":54744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":54744},{"type":"v1","addr":"172.21.15.125:6815","nonce":54744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":54744},{"type":"v1","addr":"172.21.15.125:6812","nonce":54744}]},"public_addr":"172.21.15.125:6804/54744","cluster_addr":"172.21.15.125:6808/54744","heartbeat_back_addr":"172.21.15.125:6815/54744","heartbeat_front_addr":"172.21.15.125:6812/54744","state":["exists","up"]},{"osd":5,"uuid":"a8ae81ab-36f1-4c1b-908b-4049b805a2b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":31378},{"type":"v1","addr":"172.21.15.125:6801","nonce":31378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6803","nonce":31378},{"type":"v1","addr":"172.21.15.125:6805","nonce":31378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":31378},{"type":"v1","addr":"172.21.15.125:6813","nonce":31378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6807","nonce":31378},{"type":"v1","addr":"172.21.15.125:6809","nonce":31378}]},"public_addr":"172.21.15.125:6801/31378","cluster_addr":"172.21.15.125:6805/31378","heartbeat_back_addr":"172.21.15.125:6813/31378","heartbeat_front_addr":"172.21.15.125:6809/31378","state":["exists","up"]},{"osd":6,"uuid":"e6d12493-c134-4cd9-88b5-a1e5eb2ad553","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":31379},{"type":"v1","addr":"172.21.15.125:6817","nonce":31379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":31379},{"type":"v1","addr":"172.21.15.125:6819","nonce":31379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":31379},{"type":"v1","addr":"172.21.15.125:6826","nonce":31379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":31379},{"type":"v1","addr":"172.21.15.125:6822","nonce":31379}]},"public_addr":"172.21.15.125:6817/31379","cluster_addr":"172.21.15.125:6819/31379","heartbeat_back_addr":"172.21.15.125:6826/31379","heartbeat_front_addr":"172.21.15.125:6822/31379","state":["exists","up"]},{"osd":7,"uuid":"3c4fce39-796b-492e-b19e-db661997ae80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":145,"up_from":265,"up_thru":278,"down_at":147,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":59463},{"type":"v1","addr":"172.21.15.125:6823","nonce":59463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":59463},{"type":"v1","addr":"172.21.15.125:6827","nonce":59463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":59463},{"type":"v1","addr":"172.21.15.125:6831","nonce":59463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":59463},{"type":"v1","addr":"172.21.15.125:6829","nonce":59463}]},"public_addr":"172.21.15.125:6823/59463","cluster_addr":"172.21.15.125:6827/59463","heartbeat_back_addr":"172.21.15.125:6831/59463","heartbeat_front_addr":"172.21.15.125:6829/59463","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:31.943518+0000","dead_epoch":59},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:42.554598+0000","dead_epoch":109},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:08.792704+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:09.199838+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:15.817034+0000","dead_epoch":177},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:07.780253+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:09.525009+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:38.275071+0000","dead_epoch":147}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.2","osds":[1,3,5]},{"pgid":"1.3","osds":[1,7,5]},{"pgid":"1.4","osds":[1,7,5]},{"pgid":"1.7","osds":[1,0,7]},{"pgid":"1.b","osds":[5,1,2]},{"pgid":"1.c","osds":[5,1,0]},{"pgid":"1.e","osds":[3,1,2]},{"pgid":"1.f","osds":[5,1,2]},{"pgid":"1.11","osds":[3,5,4]},{"pgid":"1.12","osds":[3,5,4]},{"pgid":"1.13","osds":[5,2,4]},{"pgid":"1.14","osds":[5,2,4]}],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} , 2020-12-18T07:52:05.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.0 flush_pg_stats Ignoring malparsed backtrace: 2020-12-18T07:52:58.334 INFO:teuthology.orchestra.run.smithi055.stdout:{ , 2020-12-18T07:52:58.334 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-18T07:52:58.334 INFO:teuthology.orchestra.run.smithi055.stdout:} Ignoring malparsed backtrace: 2020-12-18T07:53:22.125 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-12-18T07:53:34.399+0000 7f33d4590700 -1 --2- 172.21.15.55:0/4031903027 >> [v2:172.21.15.55:6803/31472,v1:172.21.15.55:6805/31472] conn(0x55c3774d2400 0x55c3776c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.55:6803/31472,v1:172.21.15.55:6805/31472] is using msgr V1 protocol , 2020-12-18T07:53:42.127 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json Ignoring malparsed backtrace: 2020-12-18T07:53:45.877 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.aQM5x5ZsNZ , 2020-12-18T07:53:45.892 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/shuzhenyi-2020-12-18_06:57:08-rados:thrash-erasure-code-wip-zhenyi-testing-2020-12-17-1108-distro-basic-smithi/5717770/data/mon.c.tgz , 2020-12-18T07:53:45.893 DEBUG:teuthology.orchestra.run.smithi055:> mktemp Ignoring malparsed backtrace: 2020-12-18T07:54:52.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service , Redirecting to /bin/systemctl restart rsyslog.service Ignoring malparsed backtrace: ceph version 16.0.0-4450-gbd745e79 (bd745e79ac867292f9946dc3bbde9d10caea3ea4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f8abd352bd4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x555c79d4d686] Ignoring malparsed backtrace: 2020-12-18T07:51:42.225 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops , 2020-12-18T07:51:42.347 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument , 2020-12-18T07:51:42.348 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-18T07:52:05.553 INFO:teuthology.orchestra.run.smithi055.stdout: , 2020-12-18T07:52:05.553 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":292,"fsid":"0b10911c-55f2-4321-98ee-e48f824cb951","created":"2020-12-18T07:45:58.166622+0000","modified":"2020-12-18T07:52:04.971452+0000","last_up_change":"2020-12-18T07:51:37.993562+0000","last_in_change":"2020-12-18T07:51:48.956269+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-18T07:46:01.658540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":256,"last_epoch_started":255,"last_epoch_clean":255,"source_version":"15'2","target_version":"15'2"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"256","last_force_op_resend_preluminous":"254","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-12-18T07:46:12.073110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-12-18T07:46:26.099909+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":124,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"c52faba0-9abc-44de-aa9b-6c251284ece1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":58,"up_from":214,"up_thru":290,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":67104},{"type":"v1","addr":"172.21.15.55:6803","nonce":67104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":67104},{"type":"v1","addr":"172.21.15.55:6805","nonce":67104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":67104},{"type":"v1","addr":"172.21.15.55:6809","nonce":67104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":67104},{"type":"v1","addr":"172.21.15.55:6807","nonce":67104}]},"public_addr":"172.21.15.55:6803/67104","cluster_addr":"172.21.15.55:6805/67104","heartbeat_back_addr":"172.21.15.55:6809/67104","heartbeat_front_addr":"172.21.15.55:6807/67104","state":["exists","up"]},{"osd":1,"uuid":"4fcb76ba-fd13-47b6-b4b5-209850e4660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":108,"up_from":221,"up_thru":291,"down_at":109,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":68578},{"type":"v1","addr":"172.21.15.55:6811","nonce":68578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":68578},{"type":"v1","addr":"172.21.15.55:6813","nonce":68578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":68578},{"type":"v1","addr":"172.21.15.55:6817","nonce":68578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":68578},{"type":"v1","addr":"172.21.15.55:6815","nonce":68578}]},"public_addr":"172.21.15.55:6811/68578","cluster_addr":"172.21.15.55:6813/68578","heartbeat_back_addr":"172.21.15.55:6817/68578","heartbeat_front_addr":"172.21.15.55:6815/68578","state":["exists","up"]},{"osd":2,"uuid":"2aabf52a-73a6-403d-b764-a86024eea78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":31473},{"type":"v1","addr":"172.21.15.55:6819","nonce":31473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":31473},{"type":"v1","addr":"172.21.15.55:6822","nonce":31473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6827","nonce":31473},{"type":"v1","addr":"172.21.15.55:6829","nonce":31473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6823","nonce":31473},{"type":"v1","addr":"172.21.15.55:6825","nonce":31473}]},"public_addr":"172.21.15.55:6819/31473","cluster_addr":"172.21.15.55:6822/31473","heartbeat_back_addr":"172.21.15.55:6829/31473","heartbeat_front_addr":"172.21.15.55:6825/31473","state":["exists","up"]},{"osd":3,"uuid":"5f2a1b95-ed63-4e0a-bb62-c935eb2ad4ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6821","nonce":31474},{"type":"v1","addr":"172.21.15.55:6824","nonce":31474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":31474},{"type":"v1","addr":"172.21.15.55:6828","nonce":31474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":31474},{"type":"v1","addr":"172.21.15.55:6833","nonce":31474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":31474},{"type":"v1","addr":"172.21.15.55:6831","nonce":31474}]},"public_addr":"172.21.15.55:6824/31474","cluster_addr":"172.21.15.55:6828/31474","heartbeat_back_addr":"172.21.15.55:6833/31474","heartbeat_front_addr":"172.21.15.55:6831/31474","state":["exists","up"]},{"osd":4,"uuid":"271d0b98-4bad-4157-bd46-d301aeeaf7d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":176,"up_from":199,"up_thru":281,"down_at":177,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":54744},{"type":"v1","addr":"172.21.15.125:6804","nonce":54744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":54744},{"type":"v1","addr":"172.21.15.125:6808","nonce":54744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":54744},{"type":"v1","addr":"172.21.15.125:6815","nonce":54744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":54744},{"type":"v1","addr":"172.21.15.125:6812","nonce":54744}]},"public_addr":"172.21.15.125:6804/54744","cluster_addr":"172.21.15.125:6808/54744","heartbeat_back_addr":"172.21.15.125:6815/54744","heartbeat_front_addr":"172.21.15.125:6812/54744","state":["exists","up"]},{"osd":5,"uuid":"a8ae81ab-36f1-4c1b-908b-4049b805a2b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":31378},{"type":"v1","addr":"172.21.15.125:6801","nonce":31378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6803","nonce":31378},{"type":"v1","addr":"172.21.15.125:6805","nonce":31378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6811","nonce":31378},{"type":"v1","addr":"172.21.15.125:6813","nonce":31378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6807","nonce":31378},{"type":"v1","addr":"172.21.15.125:6809","nonce":31378}]},"public_addr":"172.21.15.125:6801/31378","cluster_addr":"172.21.15.125:6805/31378","heartbeat_back_addr":"172.21.15.125:6813/31378","heartbeat_front_addr":"172.21.15.125:6809/31378","state":["exists","up"]},{"osd":6,"uuid":"e6d12493-c134-4cd9-88b5-a1e5eb2ad553","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":31379},{"type":"v1","addr":"172.21.15.125:6817","nonce":31379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":31379},{"type":"v1","addr":"172.21.15.125:6819","nonce":31379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":31379},{"type":"v1","addr":"172.21.15.125:6826","nonce":31379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":31379},{"type":"v1","addr":"172.21.15.125:6822","nonce":31379}]},"public_addr":"172.21.15.125:6817/31379","cluster_addr":"172.21.15.125:6819/31379","heartbeat_back_addr":"172.21.15.125:6826/31379","heartbeat_front_addr":"172.21.15.125:6822/31379","state":["exists","up"]},{"osd":7,"uuid":"3c4fce39-796b-492e-b19e-db661997ae80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":145,"up_from":265,"up_thru":278,"down_at":147,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6821","nonce":59463},{"type":"v1","addr":"172.21.15.125:6823","nonce":59463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6825","nonce":59463},{"type":"v1","addr":"172.21.15.125:6827","nonce":59463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":59463},{"type":"v1","addr":"172.21.15.125:6831","nonce":59463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":59463},{"type":"v1","addr":"172.21.15.125:6829","nonce":59463}]},"public_addr":"172.21.15.125:6823/59463","cluster_addr":"172.21.15.125:6827/59463","heartbeat_back_addr":"172.21.15.125:6831/59463","heartbeat_front_addr":"172.21.15.125:6829/59463","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:31.943518+0000","dead_epoch":59},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:42.554598+0000","dead_epoch":109},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:08.792704+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:09.199838+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:15.817034+0000","dead_epoch":177},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:07.780253+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:09.525009+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-12-18T07:51:38.275071+0000","dead_epoch":147}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.2","osds":[1,3,5]},{"pgid":"1.3","osds":[1,7,5]},{"pgid":"1.4","osds":[1,7,5]},{"pgid":"1.7","osds":[1,0,7]},{"pgid":"1.b","osds":[5,1,2]},{"pgid":"1.c","osds":[5,1,0]},{"pgid":"1.e","osds":[3,1,2]},{"pgid":"1.f","osds":[5,1,2]},{"pgid":"1.11","osds":[3,5,4]},{"pgid":"1.12","osds":[3,5,4]},{"pgid":"1.13","osds":[5,2,4]},{"pgid":"1.14","osds":[5,2,4]}],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} , 2020-12-18T07:52:05.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.0 flush_pg_stats Ignoring malparsed backtrace: 2020-12-18T07:52:58.334 INFO:teuthology.orchestra.run.smithi055.stdout:{ , 2020-12-18T07:52:58.334 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-12-18T07:52:58.334 INFO:teuthology.orchestra.run.smithi055.stdout:} Ignoring malparsed backtrace: 2020-12-18T07:53:22.125 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-12-18T07:53:34.399+0000 7f33d4590700 -1 --2- 172.21.15.55:0/4031903027 >> [v2:172.21.15.55:6803/31472,v1:172.21.15.55:6805/31472] conn(0x55c3774d2400 0x55c3776c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.55:6803/31472,v1:172.21.15.55:6805/31472] is using msgr V1 protocol , 2020-12-18T07:53:42.127 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json Ignoring malparsed backtrace: 2020-12-18T07:53:45.877 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.aQM5x5ZsNZ , 2020-12-18T07:53:45.892 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/shuzhenyi-2020-12-18_06:57:08-rados:thrash-erasure-code-wip-zhenyi-testing-2020-12-17-1108-distro-basic-smithi/5717770/data/mon.c.tgz , 2020-12-18T07:53:45.893 DEBUG:teuthology.orchestra.run.smithi055:> mktemp Ignoring malparsed backtrace: 2020-12-18T07:54:52.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , Redirecting to /bin/systemctl restart rsyslog.service , Redirecting to /bin/systemctl restart rsyslog.service Found 2 distinct failure reasons Dead: b'2020-12-18T19:30:00.402 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['5717768'] suites: ['clusters/{fixed-2', 'fast/normal', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 0 --op append_excl 50 --pool unique_pool_0' 1 jobs: ['5717770'] suites: ['clusters/{fixed-2', 'fast/fast', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-small-objects-balanced}']