2021-03-07T13:37:34.144 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-07T13:37:34.146 INFO:teuthology.results:subset: '989/99999' 2021-03-07T13:37:34.146 INFO:teuthology.results:seed: '2477' 2021-03-07T13:58:36.532 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-07T13:58:37.656 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-08T01:48:30.945 INFO:teuthology.results:Tests finished! gathering results... 2021-03-08T01:48:36.201 INFO:scrape:Found 26 jobs 2021-03-08T01:48:37.762 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1695-gd5f415b9 (d5f415b9b67a6d66ef39db91322ef188bb9f833e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f3f7e117fc8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2761e2) [0x7f3f7e1181e2] 2021-03-08T01:48:37.763 WARNING:scrape:Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/git.ceph.com_ceph-c_d5f415b9b67a6d66ef39db91322ef188bb9f833e/qa/tasks/rados.py", line 265, in thread 2021-03-08T01:48:37.845 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:43.560 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":404,"fsid":"43481236-7a4d-436b-86b0-491b76d17c89","created":"2021-03-07T13:56:48.736434+0000","modified":"2021-03-07T14:03:42.872342+0000","last_up_change":"2021-03-07T14:02:19.639180+0000","last_in_change":"2021-03-07T14:03:13.676195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-07T13:56:59.279245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"358","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"149","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-07T13:57:01.712160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"111","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-07T13:57:16.685229+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":6,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"372","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":323,"snap_epoch":372,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"jerasure21profile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"b3ff6f99-f475-4496-9836-f0813e4412dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":33,"up_from":127,"up_thru":379,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":51471},{"type":"v1","addr":"172.21.15.59:6801","nonce":51471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":51471},{"type":"v1","addr":"172.21.15.59:6803","nonce":51471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":51471},{"type":"v1","addr":"172.21.15.59:6810","nonce":51471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":51471},{"type":"v1","addr":"172.21.15.59:6806","nonce":51471}]},"public_addr":"172.21.15.59:6801/51471","cluster_addr":"172.21.15.59:6803/51471","heartbeat_back_addr":"172.21.15.59:6810/51471","heartbeat_front_addr":"172.21.15.59:6806/51471","state":["exists","up"]},{"osd":1,"uuid":"7c282045-3f47-4c2f-94bb-1f0be7efd0d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":33226},{"type":"v1","addr":"172.21.15.59:6807","nonce":33226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":33226},{"type":"v1","addr":"172.21.15.59:6811","nonce":33226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":33226},{"type":"v1","addr":"172.21.15.59:6815","nonce":33226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":33226},{"type":"v1","addr":"172.21.15.59:6813","nonce":33226}]},"public_addr":"172.21.15.59:6807/33226","cluster_addr":"172.21.15.59:6811/33226","heartbeat_back_addr":"172.21.15.59:6815/33226","heartbeat_front_addr":"172.21.15.59:6813/33226","state":["exists","up"]},{"osd":2,"uuid":"4c89c1b2-aade-402e-b057-44297690cbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":377,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6817","nonce":33227},{"type":"v1","addr":"172.21.15.59:6818","nonce":33227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":33227},{"type":"v1","addr":"172.21.15.59:6823","nonce":33227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":33227},{"type":"v1","addr":"172.21.15.59:6831","nonce":33227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":33227},{"type":"v1","addr":"172.21.15.59:6827","nonce":33227}]},"public_addr":"172.21.15.59:6818/33227","cluster_addr":"172.21.15.59:6823/33227","heartbeat_back_addr":"172.21.15.59:6831/33227","heartbeat_front_addr":"172.21.15.59:6827/33227","state":["exists","up"]},{"osd":3,"uuid":"7b03b655-a408-4331-bc3b-87fc476d7fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":33228},{"type":"v1","addr":"172.21.15.59:6819","nonce":33228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":33228},{"type":"v1","addr":"172.21.15.59:6822","nonce":33228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":33228},{"type":"v1","addr":"172.21.15.59:6830","nonce":33228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":33228},{"type":"v1","addr":"172.21.15.59:6826","nonce":33228}]},"public_addr":"172.21.15.59:6819/33228","cluster_addr":"172.21.15.59:6822/33228","heartbeat_back_addr":"172.21.15.59:6830/33228","heartbeat_front_addr":"172.21.15.59:6826/33228","state":["exists","up"]},{"osd":4,"uuid":"f8d3bc78-4601-40c2-9e29-bc29cb49eb94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":33087},{"type":"v1","addr":"172.21.15.134:6801","nonce":33087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":33087},{"type":"v1","addr":"172.21.15.134:6803","nonce":33087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":33087},{"type":"v1","addr":"172.21.15.134:6807","nonce":33087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":33087},{"type":"v1","addr":"172.21.15.134:6805","nonce":33087}]},"public_addr":"172.21.15.134:6801/33087","cluster_addr":"172.21.15.134:6803/33087","heartbeat_back_addr":"172.21.15.134:6807/33087","heartbeat_front_addr":"172.21.15.134:6805/33087","state":["exists","up"]},{"osd":5,"uuid":"839c9ef0-c471-43b4-8fa4-5f993c2ec671","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":297,"up_from":321,"up_thru":378,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":51924},{"type":"v1","addr":"172.21.15.134:6817","nonce":51924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":51924},{"type":"v1","addr":"172.21.15.134:6819","nonce":51924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6823","nonce":51924},{"type":"v1","addr":"172.21.15.134:6825","nonce":51924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":51924},{"type":"v1","addr":"172.21.15.134:6821","nonce":51924}]},"public_addr":"172.21.15.134:6817/51924","cluster_addr":"172.21.15.134:6819/51924","heartbeat_back_addr":"172.21.15.134:6825/51924","heartbeat_front_addr":"172.21.15.134:6821/51924","state":["exists","up"]},{"osd":6,"uuid":"93af4222-cb52-4fcc-abbe-42c7df688f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":390,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":33088},{"type":"v1","addr":"172.21.15.134:6809","nonce":33088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":33088},{"type":"v1","addr":"172.21.15.134:6811","nonce":33088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":33088},{"type":"v1","addr":"172.21.15.134:6815","nonce":33088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":33088},{"type":"v1","addr":"172.21.15.134:6813","nonce":33088}]},"public_addr":"172.21.15.134:6809/33088","cluster_addr":"172.21.15.134:6811/33088","heartbeat_back_addr":"172.21.15.134:6815/33088","heartbeat_front_addr":"172.21.15.134:6813/33088","state":["exists","up"]},{"osd":7,"uuid":"7325f570-a0fc-468d-8b63-11802b68686c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":33118},{"type":"v1","addr":"172.21.15.134:6824","nonce":33118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":33118},{"type":"v1","addr":"172.21.15.134:6827","nonce":33118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":33118},{"type":"v1","addr":"172.21.15.134:6831","nonce":33118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":33118},{"type":"v1","addr":"172.21.15.134:6829","nonce":33118}]},"public_addr":"172.21.15.134:6824/33118","cluster_addr":"172.21.15.134:6827/33118","heartbeat_back_addr":"172.21.15.134:6831/33118","heartbeat_front_addr":"172.21.15.134:6829/33118","state":["exists","up"]},{"osd":8,"uuid":"e59c0b86-90a7-4fbf-b8e5-bb718050f86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":33587},{"type":"v1","addr":"172.21.15.141:6801","nonce":33587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":33587},{"type":"v1","addr":"172.21.15.141:6803","nonce":33587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":33587},{"type":"v1","addr":"172.21.15.141:6807","nonce":33587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":33587},{"type":"v1","addr":"172.21.15.141:6805","nonce":33587}]},"public_addr":"172.21.15.141:6801/33587","cluster_addr":"172.21.15.141:6803/33587","heartbeat_back_addr":"172.21.15.141:6807/33587","heartbeat_front_addr":"172.21.15.141:6805/33587","state":["exists","up"]},{"osd":9,"uuid":"82a3016c-d9a2-4486-90de-e79a3011c56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":373,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6809","nonce":33588},{"type":"v1","addr":"172.21.15.141:6811","nonce":33588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6813","nonce":33588},{"type":"v1","addr":"172.21.15.141:6815","nonce":33588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6821","nonce":33588},{"type":"v1","addr":"172.21.15.141:6823","nonce":33588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6817","nonce":33588},{"type":"v1","addr":"172.21.15.141:6819","nonce":33588}]},"public_addr":"172.21.15.141:6811/33588","cluster_addr":"172.21.15.141:6815/33588","heartbeat_back_addr":"172.21.15.141:6823/33588","heartbeat_front_addr":"172.21.15.141:6819/33588","state":["exists","up"]},{"osd":10,"uuid":"d24ccde9-6e88-4fd6-b95f-0682b0864c0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":33589},{"type":"v1","addr":"172.21.15.141:6810","nonce":33589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":33589},{"type":"v1","addr":"172.21.15.141:6814","nonce":33589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":33589},{"type":"v1","addr":"172.21.15.141:6822","nonce":33589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":33589},{"type":"v1","addr":"172.21.15.141:6818","nonce":33589}]},"public_addr":"172.21.15.141:6810/33589","cluster_addr":"172.21.15.141:6814/33589","heartbeat_back_addr":"172.21.15.141:6822/33589","heartbeat_front_addr":"172.21.15.141:6818/33589","state":["exists","up"]},{"osd":11,"uuid":"da7db835-7c2d-4329-8f98-5a9164fa3450","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":33590},{"type":"v1","addr":"172.21.15.141:6825","nonce":33590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":33590},{"type":"v1","addr":"172.21.15.141:6827","nonce":33590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":33590},{"type":"v1","addr":"172.21.15.141:6831","nonce":33590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":33590},{"type":"v1","addr":"172.21.15.141:6829","nonce":33590}]},"public_addr":"172.21.15.141:6825/33590","cluster_addr":"172.21.15.141:6827/33590","heartbeat_back_addr":"172.21.15.141:6831/33590","heartbeat_front_addr":"172.21.15.141:6829/33590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:01.845974+0000","dead_epoch":34},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.121271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.669451+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.354991+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.093103+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:19.770700+0000","dead_epoch":298},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.594108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.799685+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.941830+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.824769+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.002341+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.753674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[3,4]},{"pgid":"2.b","osds":[10,11]},{"pgid":"3.c","osds":[11,10,2,3,4,6]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"jerasure21profile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"4","m":"2","name":"jerasure21profile","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-03-07T14:03:43.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats , 2021-03-07T14:03:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:545460846650 2021-03-08T01:48:37.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:53.880 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2021-03-07T14:03:53.881 INFO:teuthology.orchestra.run.smithi059.stdout:} , 2021-03-07T14:03:53.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2021-03-08T01:48:37.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2021-03-07T14:03:41.804607+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=41, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2021-03-07T14:03:48.250901+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=48, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.044 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2021-03-07T14:02:50.751738+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=50, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) 2021-03-08T01:48:37.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:04:40.867 DEBUG:tasks.ceph.mon.c:waiting for process to exit , 2021-03-07T14:04:40.867 INFO:teuthology.orchestra.run:waiting for 300 , 2021-03-07T14:04:40.877 INFO:tasks.ceph.mon.c:Stopped 2021-03-08T01:48:37.850 WARNING:scrape:Ignoring malparsed backtrace: proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() 2021-03-08T01:48:37.851 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2021-03-08T01:48:38.150 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1695-gd5f415b9 (d5f415b9b67a6d66ef39db91322ef188bb9f833e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f3f7e117fc8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2761e2) [0x7f3f7e1181e2] 2021-03-08T01:48:38.151 WARNING:scrape:Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/git.ceph.com_ceph-c_d5f415b9b67a6d66ef39db91322ef188bb9f833e/qa/tasks/rados.py", line 265, in thread 2021-03-08T01:48:38.153 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:43.560 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":404,"fsid":"43481236-7a4d-436b-86b0-491b76d17c89","created":"2021-03-07T13:56:48.736434+0000","modified":"2021-03-07T14:03:42.872342+0000","last_up_change":"2021-03-07T14:02:19.639180+0000","last_in_change":"2021-03-07T14:03:13.676195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-07T13:56:59.279245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"358","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"149","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-07T13:57:01.712160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"111","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-07T13:57:16.685229+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":6,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"372","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":323,"snap_epoch":372,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"jerasure21profile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"b3ff6f99-f475-4496-9836-f0813e4412dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":33,"up_from":127,"up_thru":379,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":51471},{"type":"v1","addr":"172.21.15.59:6801","nonce":51471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":51471},{"type":"v1","addr":"172.21.15.59:6803","nonce":51471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":51471},{"type":"v1","addr":"172.21.15.59:6810","nonce":51471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":51471},{"type":"v1","addr":"172.21.15.59:6806","nonce":51471}]},"public_addr":"172.21.15.59:6801/51471","cluster_addr":"172.21.15.59:6803/51471","heartbeat_back_addr":"172.21.15.59:6810/51471","heartbeat_front_addr":"172.21.15.59:6806/51471","state":["exists","up"]},{"osd":1,"uuid":"7c282045-3f47-4c2f-94bb-1f0be7efd0d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":33226},{"type":"v1","addr":"172.21.15.59:6807","nonce":33226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":33226},{"type":"v1","addr":"172.21.15.59:6811","nonce":33226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":33226},{"type":"v1","addr":"172.21.15.59:6815","nonce":33226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":33226},{"type":"v1","addr":"172.21.15.59:6813","nonce":33226}]},"public_addr":"172.21.15.59:6807/33226","cluster_addr":"172.21.15.59:6811/33226","heartbeat_back_addr":"172.21.15.59:6815/33226","heartbeat_front_addr":"172.21.15.59:6813/33226","state":["exists","up"]},{"osd":2,"uuid":"4c89c1b2-aade-402e-b057-44297690cbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":377,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6817","nonce":33227},{"type":"v1","addr":"172.21.15.59:6818","nonce":33227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":33227},{"type":"v1","addr":"172.21.15.59:6823","nonce":33227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":33227},{"type":"v1","addr":"172.21.15.59:6831","nonce":33227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":33227},{"type":"v1","addr":"172.21.15.59:6827","nonce":33227}]},"public_addr":"172.21.15.59:6818/33227","cluster_addr":"172.21.15.59:6823/33227","heartbeat_back_addr":"172.21.15.59:6831/33227","heartbeat_front_addr":"172.21.15.59:6827/33227","state":["exists","up"]},{"osd":3,"uuid":"7b03b655-a408-4331-bc3b-87fc476d7fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":33228},{"type":"v1","addr":"172.21.15.59:6819","nonce":33228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":33228},{"type":"v1","addr":"172.21.15.59:6822","nonce":33228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":33228},{"type":"v1","addr":"172.21.15.59:6830","nonce":33228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":33228},{"type":"v1","addr":"172.21.15.59:6826","nonce":33228}]},"public_addr":"172.21.15.59:6819/33228","cluster_addr":"172.21.15.59:6822/33228","heartbeat_back_addr":"172.21.15.59:6830/33228","heartbeat_front_addr":"172.21.15.59:6826/33228","state":["exists","up"]},{"osd":4,"uuid":"f8d3bc78-4601-40c2-9e29-bc29cb49eb94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":33087},{"type":"v1","addr":"172.21.15.134:6801","nonce":33087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":33087},{"type":"v1","addr":"172.21.15.134:6803","nonce":33087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":33087},{"type":"v1","addr":"172.21.15.134:6807","nonce":33087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":33087},{"type":"v1","addr":"172.21.15.134:6805","nonce":33087}]},"public_addr":"172.21.15.134:6801/33087","cluster_addr":"172.21.15.134:6803/33087","heartbeat_back_addr":"172.21.15.134:6807/33087","heartbeat_front_addr":"172.21.15.134:6805/33087","state":["exists","up"]},{"osd":5,"uuid":"839c9ef0-c471-43b4-8fa4-5f993c2ec671","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":297,"up_from":321,"up_thru":378,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":51924},{"type":"v1","addr":"172.21.15.134:6817","nonce":51924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":51924},{"type":"v1","addr":"172.21.15.134:6819","nonce":51924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6823","nonce":51924},{"type":"v1","addr":"172.21.15.134:6825","nonce":51924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":51924},{"type":"v1","addr":"172.21.15.134:6821","nonce":51924}]},"public_addr":"172.21.15.134:6817/51924","cluster_addr":"172.21.15.134:6819/51924","heartbeat_back_addr":"172.21.15.134:6825/51924","heartbeat_front_addr":"172.21.15.134:6821/51924","state":["exists","up"]},{"osd":6,"uuid":"93af4222-cb52-4fcc-abbe-42c7df688f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":390,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":33088},{"type":"v1","addr":"172.21.15.134:6809","nonce":33088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":33088},{"type":"v1","addr":"172.21.15.134:6811","nonce":33088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":33088},{"type":"v1","addr":"172.21.15.134:6815","nonce":33088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":33088},{"type":"v1","addr":"172.21.15.134:6813","nonce":33088}]},"public_addr":"172.21.15.134:6809/33088","cluster_addr":"172.21.15.134:6811/33088","heartbeat_back_addr":"172.21.15.134:6815/33088","heartbeat_front_addr":"172.21.15.134:6813/33088","state":["exists","up"]},{"osd":7,"uuid":"7325f570-a0fc-468d-8b63-11802b68686c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":33118},{"type":"v1","addr":"172.21.15.134:6824","nonce":33118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":33118},{"type":"v1","addr":"172.21.15.134:6827","nonce":33118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":33118},{"type":"v1","addr":"172.21.15.134:6831","nonce":33118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":33118},{"type":"v1","addr":"172.21.15.134:6829","nonce":33118}]},"public_addr":"172.21.15.134:6824/33118","cluster_addr":"172.21.15.134:6827/33118","heartbeat_back_addr":"172.21.15.134:6831/33118","heartbeat_front_addr":"172.21.15.134:6829/33118","state":["exists","up"]},{"osd":8,"uuid":"e59c0b86-90a7-4fbf-b8e5-bb718050f86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":33587},{"type":"v1","addr":"172.21.15.141:6801","nonce":33587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":33587},{"type":"v1","addr":"172.21.15.141:6803","nonce":33587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":33587},{"type":"v1","addr":"172.21.15.141:6807","nonce":33587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":33587},{"type":"v1","addr":"172.21.15.141:6805","nonce":33587}]},"public_addr":"172.21.15.141:6801/33587","cluster_addr":"172.21.15.141:6803/33587","heartbeat_back_addr":"172.21.15.141:6807/33587","heartbeat_front_addr":"172.21.15.141:6805/33587","state":["exists","up"]},{"osd":9,"uuid":"82a3016c-d9a2-4486-90de-e79a3011c56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":373,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6809","nonce":33588},{"type":"v1","addr":"172.21.15.141:6811","nonce":33588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6813","nonce":33588},{"type":"v1","addr":"172.21.15.141:6815","nonce":33588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6821","nonce":33588},{"type":"v1","addr":"172.21.15.141:6823","nonce":33588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6817","nonce":33588},{"type":"v1","addr":"172.21.15.141:6819","nonce":33588}]},"public_addr":"172.21.15.141:6811/33588","cluster_addr":"172.21.15.141:6815/33588","heartbeat_back_addr":"172.21.15.141:6823/33588","heartbeat_front_addr":"172.21.15.141:6819/33588","state":["exists","up"]},{"osd":10,"uuid":"d24ccde9-6e88-4fd6-b95f-0682b0864c0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":33589},{"type":"v1","addr":"172.21.15.141:6810","nonce":33589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":33589},{"type":"v1","addr":"172.21.15.141:6814","nonce":33589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":33589},{"type":"v1","addr":"172.21.15.141:6822","nonce":33589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":33589},{"type":"v1","addr":"172.21.15.141:6818","nonce":33589}]},"public_addr":"172.21.15.141:6810/33589","cluster_addr":"172.21.15.141:6814/33589","heartbeat_back_addr":"172.21.15.141:6822/33589","heartbeat_front_addr":"172.21.15.141:6818/33589","state":["exists","up"]},{"osd":11,"uuid":"da7db835-7c2d-4329-8f98-5a9164fa3450","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":33590},{"type":"v1","addr":"172.21.15.141:6825","nonce":33590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":33590},{"type":"v1","addr":"172.21.15.141:6827","nonce":33590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":33590},{"type":"v1","addr":"172.21.15.141:6831","nonce":33590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":33590},{"type":"v1","addr":"172.21.15.141:6829","nonce":33590}]},"public_addr":"172.21.15.141:6825/33590","cluster_addr":"172.21.15.141:6827/33590","heartbeat_back_addr":"172.21.15.141:6831/33590","heartbeat_front_addr":"172.21.15.141:6829/33590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:01.845974+0000","dead_epoch":34},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.121271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.669451+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.354991+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.093103+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:19.770700+0000","dead_epoch":298},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.594108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.799685+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.941830+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.824769+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.002341+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.753674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[3,4]},{"pgid":"2.b","osds":[10,11]},{"pgid":"3.c","osds":[11,10,2,3,4,6]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"jerasure21profile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"4","m":"2","name":"jerasure21profile","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-03-07T14:03:43.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats , 2021-03-07T14:03:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:545460846650 2021-03-08T01:48:38.155 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:53.880 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2021-03-07T14:03:53.881 INFO:teuthology.orchestra.run.smithi059.stdout:} , 2021-03-07T14:03:53.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2021-03-08T01:48:38.156 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2021-03-07T14:03:41.804607+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=41, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2021-03-07T14:03:48.250901+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=48, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.044 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2021-03-07T14:02:50.751738+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=50, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) 2021-03-08T01:48:38.157 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:04:40.867 DEBUG:tasks.ceph.mon.c:waiting for process to exit , 2021-03-07T14:04:40.867 INFO:teuthology.orchestra.run:waiting for 300 , 2021-03-07T14:04:40.877 INFO:tasks.ceph.mon.c:Stopped 2021-03-08T01:48:38.157 WARNING:scrape:Ignoring malparsed backtrace: proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() 2021-03-08T01:48:38.158 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2021-03-08T01:48:38.475 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1695-gd5f415b9 (d5f415b9b67a6d66ef39db91322ef188bb9f833e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f3f7e117fc8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2761e2) [0x7f3f7e1181e2] 2021-03-08T01:48:38.476 WARNING:scrape:Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/git.ceph.com_ceph-c_d5f415b9b67a6d66ef39db91322ef188bb9f833e/qa/tasks/rados.py", line 265, in thread 2021-03-08T01:48:38.479 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:43.560 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":404,"fsid":"43481236-7a4d-436b-86b0-491b76d17c89","created":"2021-03-07T13:56:48.736434+0000","modified":"2021-03-07T14:03:42.872342+0000","last_up_change":"2021-03-07T14:02:19.639180+0000","last_in_change":"2021-03-07T14:03:13.676195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-07T13:56:59.279245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"358","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"149","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-07T13:57:01.712160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"111","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-07T13:57:16.685229+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":6,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"372","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":323,"snap_epoch":372,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"jerasure21profile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"b3ff6f99-f475-4496-9836-f0813e4412dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":33,"up_from":127,"up_thru":379,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":51471},{"type":"v1","addr":"172.21.15.59:6801","nonce":51471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":51471},{"type":"v1","addr":"172.21.15.59:6803","nonce":51471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":51471},{"type":"v1","addr":"172.21.15.59:6810","nonce":51471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":51471},{"type":"v1","addr":"172.21.15.59:6806","nonce":51471}]},"public_addr":"172.21.15.59:6801/51471","cluster_addr":"172.21.15.59:6803/51471","heartbeat_back_addr":"172.21.15.59:6810/51471","heartbeat_front_addr":"172.21.15.59:6806/51471","state":["exists","up"]},{"osd":1,"uuid":"7c282045-3f47-4c2f-94bb-1f0be7efd0d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":33226},{"type":"v1","addr":"172.21.15.59:6807","nonce":33226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":33226},{"type":"v1","addr":"172.21.15.59:6811","nonce":33226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":33226},{"type":"v1","addr":"172.21.15.59:6815","nonce":33226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":33226},{"type":"v1","addr":"172.21.15.59:6813","nonce":33226}]},"public_addr":"172.21.15.59:6807/33226","cluster_addr":"172.21.15.59:6811/33226","heartbeat_back_addr":"172.21.15.59:6815/33226","heartbeat_front_addr":"172.21.15.59:6813/33226","state":["exists","up"]},{"osd":2,"uuid":"4c89c1b2-aade-402e-b057-44297690cbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":377,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6817","nonce":33227},{"type":"v1","addr":"172.21.15.59:6818","nonce":33227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":33227},{"type":"v1","addr":"172.21.15.59:6823","nonce":33227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":33227},{"type":"v1","addr":"172.21.15.59:6831","nonce":33227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":33227},{"type":"v1","addr":"172.21.15.59:6827","nonce":33227}]},"public_addr":"172.21.15.59:6818/33227","cluster_addr":"172.21.15.59:6823/33227","heartbeat_back_addr":"172.21.15.59:6831/33227","heartbeat_front_addr":"172.21.15.59:6827/33227","state":["exists","up"]},{"osd":3,"uuid":"7b03b655-a408-4331-bc3b-87fc476d7fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":33228},{"type":"v1","addr":"172.21.15.59:6819","nonce":33228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":33228},{"type":"v1","addr":"172.21.15.59:6822","nonce":33228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":33228},{"type":"v1","addr":"172.21.15.59:6830","nonce":33228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":33228},{"type":"v1","addr":"172.21.15.59:6826","nonce":33228}]},"public_addr":"172.21.15.59:6819/33228","cluster_addr":"172.21.15.59:6822/33228","heartbeat_back_addr":"172.21.15.59:6830/33228","heartbeat_front_addr":"172.21.15.59:6826/33228","state":["exists","up"]},{"osd":4,"uuid":"f8d3bc78-4601-40c2-9e29-bc29cb49eb94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":33087},{"type":"v1","addr":"172.21.15.134:6801","nonce":33087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":33087},{"type":"v1","addr":"172.21.15.134:6803","nonce":33087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":33087},{"type":"v1","addr":"172.21.15.134:6807","nonce":33087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":33087},{"type":"v1","addr":"172.21.15.134:6805","nonce":33087}]},"public_addr":"172.21.15.134:6801/33087","cluster_addr":"172.21.15.134:6803/33087","heartbeat_back_addr":"172.21.15.134:6807/33087","heartbeat_front_addr":"172.21.15.134:6805/33087","state":["exists","up"]},{"osd":5,"uuid":"839c9ef0-c471-43b4-8fa4-5f993c2ec671","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":297,"up_from":321,"up_thru":378,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":51924},{"type":"v1","addr":"172.21.15.134:6817","nonce":51924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":51924},{"type":"v1","addr":"172.21.15.134:6819","nonce":51924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6823","nonce":51924},{"type":"v1","addr":"172.21.15.134:6825","nonce":51924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":51924},{"type":"v1","addr":"172.21.15.134:6821","nonce":51924}]},"public_addr":"172.21.15.134:6817/51924","cluster_addr":"172.21.15.134:6819/51924","heartbeat_back_addr":"172.21.15.134:6825/51924","heartbeat_front_addr":"172.21.15.134:6821/51924","state":["exists","up"]},{"osd":6,"uuid":"93af4222-cb52-4fcc-abbe-42c7df688f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":390,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":33088},{"type":"v1","addr":"172.21.15.134:6809","nonce":33088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":33088},{"type":"v1","addr":"172.21.15.134:6811","nonce":33088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":33088},{"type":"v1","addr":"172.21.15.134:6815","nonce":33088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":33088},{"type":"v1","addr":"172.21.15.134:6813","nonce":33088}]},"public_addr":"172.21.15.134:6809/33088","cluster_addr":"172.21.15.134:6811/33088","heartbeat_back_addr":"172.21.15.134:6815/33088","heartbeat_front_addr":"172.21.15.134:6813/33088","state":["exists","up"]},{"osd":7,"uuid":"7325f570-a0fc-468d-8b63-11802b68686c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":33118},{"type":"v1","addr":"172.21.15.134:6824","nonce":33118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":33118},{"type":"v1","addr":"172.21.15.134:6827","nonce":33118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":33118},{"type":"v1","addr":"172.21.15.134:6831","nonce":33118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":33118},{"type":"v1","addr":"172.21.15.134:6829","nonce":33118}]},"public_addr":"172.21.15.134:6824/33118","cluster_addr":"172.21.15.134:6827/33118","heartbeat_back_addr":"172.21.15.134:6831/33118","heartbeat_front_addr":"172.21.15.134:6829/33118","state":["exists","up"]},{"osd":8,"uuid":"e59c0b86-90a7-4fbf-b8e5-bb718050f86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":33587},{"type":"v1","addr":"172.21.15.141:6801","nonce":33587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":33587},{"type":"v1","addr":"172.21.15.141:6803","nonce":33587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":33587},{"type":"v1","addr":"172.21.15.141:6807","nonce":33587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":33587},{"type":"v1","addr":"172.21.15.141:6805","nonce":33587}]},"public_addr":"172.21.15.141:6801/33587","cluster_addr":"172.21.15.141:6803/33587","heartbeat_back_addr":"172.21.15.141:6807/33587","heartbeat_front_addr":"172.21.15.141:6805/33587","state":["exists","up"]},{"osd":9,"uuid":"82a3016c-d9a2-4486-90de-e79a3011c56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":373,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6809","nonce":33588},{"type":"v1","addr":"172.21.15.141:6811","nonce":33588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6813","nonce":33588},{"type":"v1","addr":"172.21.15.141:6815","nonce":33588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6821","nonce":33588},{"type":"v1","addr":"172.21.15.141:6823","nonce":33588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6817","nonce":33588},{"type":"v1","addr":"172.21.15.141:6819","nonce":33588}]},"public_addr":"172.21.15.141:6811/33588","cluster_addr":"172.21.15.141:6815/33588","heartbeat_back_addr":"172.21.15.141:6823/33588","heartbeat_front_addr":"172.21.15.141:6819/33588","state":["exists","up"]},{"osd":10,"uuid":"d24ccde9-6e88-4fd6-b95f-0682b0864c0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":33589},{"type":"v1","addr":"172.21.15.141:6810","nonce":33589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":33589},{"type":"v1","addr":"172.21.15.141:6814","nonce":33589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":33589},{"type":"v1","addr":"172.21.15.141:6822","nonce":33589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":33589},{"type":"v1","addr":"172.21.15.141:6818","nonce":33589}]},"public_addr":"172.21.15.141:6810/33589","cluster_addr":"172.21.15.141:6814/33589","heartbeat_back_addr":"172.21.15.141:6822/33589","heartbeat_front_addr":"172.21.15.141:6818/33589","state":["exists","up"]},{"osd":11,"uuid":"da7db835-7c2d-4329-8f98-5a9164fa3450","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":33590},{"type":"v1","addr":"172.21.15.141:6825","nonce":33590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":33590},{"type":"v1","addr":"172.21.15.141:6827","nonce":33590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":33590},{"type":"v1","addr":"172.21.15.141:6831","nonce":33590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":33590},{"type":"v1","addr":"172.21.15.141:6829","nonce":33590}]},"public_addr":"172.21.15.141:6825/33590","cluster_addr":"172.21.15.141:6827/33590","heartbeat_back_addr":"172.21.15.141:6831/33590","heartbeat_front_addr":"172.21.15.141:6829/33590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:01.845974+0000","dead_epoch":34},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.121271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.669451+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.354991+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.093103+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:19.770700+0000","dead_epoch":298},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.594108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.799685+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.941830+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.824769+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.002341+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.753674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[3,4]},{"pgid":"2.b","osds":[10,11]},{"pgid":"3.c","osds":[11,10,2,3,4,6]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"jerasure21profile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"4","m":"2","name":"jerasure21profile","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-03-07T14:03:43.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats , 2021-03-07T14:03:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:545460846650 2021-03-08T01:48:38.480 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:53.880 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2021-03-07T14:03:53.881 INFO:teuthology.orchestra.run.smithi059.stdout:} , 2021-03-07T14:03:53.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2021-03-08T01:48:38.482 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2021-03-07T14:03:41.804607+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=41, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2021-03-07T14:03:48.250901+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=48, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.044 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2021-03-07T14:02:50.751738+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=50, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) 2021-03-08T01:48:38.483 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:04:40.867 DEBUG:tasks.ceph.mon.c:waiting for process to exit , 2021-03-07T14:04:40.867 INFO:teuthology.orchestra.run:waiting for 300 , 2021-03-07T14:04:40.877 INFO:tasks.ceph.mon.c:Stopped 2021-03-08T01:48:38.483 WARNING:scrape:Ignoring malparsed backtrace: proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() 2021-03-08T01:48:38.484 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2021-03-08T01:48:38.805 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1695-gd5f415b9 (d5f415b9b67a6d66ef39db91322ef188bb9f833e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f3f7e117fc8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2761e2) [0x7f3f7e1181e2] 2021-03-08T01:48:38.806 WARNING:scrape:Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/git.ceph.com_ceph-c_d5f415b9b67a6d66ef39db91322ef188bb9f833e/qa/tasks/rados.py", line 265, in thread 2021-03-08T01:48:38.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:43.560 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":404,"fsid":"43481236-7a4d-436b-86b0-491b76d17c89","created":"2021-03-07T13:56:48.736434+0000","modified":"2021-03-07T14:03:42.872342+0000","last_up_change":"2021-03-07T14:02:19.639180+0000","last_in_change":"2021-03-07T14:03:13.676195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-07T13:56:59.279245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"358","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"149","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-07T13:57:01.712160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"111","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-07T13:57:16.685229+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":6,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"372","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":323,"snap_epoch":372,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"jerasure21profile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"b3ff6f99-f475-4496-9836-f0813e4412dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":33,"up_from":127,"up_thru":379,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":51471},{"type":"v1","addr":"172.21.15.59:6801","nonce":51471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":51471},{"type":"v1","addr":"172.21.15.59:6803","nonce":51471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":51471},{"type":"v1","addr":"172.21.15.59:6810","nonce":51471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":51471},{"type":"v1","addr":"172.21.15.59:6806","nonce":51471}]},"public_addr":"172.21.15.59:6801/51471","cluster_addr":"172.21.15.59:6803/51471","heartbeat_back_addr":"172.21.15.59:6810/51471","heartbeat_front_addr":"172.21.15.59:6806/51471","state":["exists","up"]},{"osd":1,"uuid":"7c282045-3f47-4c2f-94bb-1f0be7efd0d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":33226},{"type":"v1","addr":"172.21.15.59:6807","nonce":33226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":33226},{"type":"v1","addr":"172.21.15.59:6811","nonce":33226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":33226},{"type":"v1","addr":"172.21.15.59:6815","nonce":33226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":33226},{"type":"v1","addr":"172.21.15.59:6813","nonce":33226}]},"public_addr":"172.21.15.59:6807/33226","cluster_addr":"172.21.15.59:6811/33226","heartbeat_back_addr":"172.21.15.59:6815/33226","heartbeat_front_addr":"172.21.15.59:6813/33226","state":["exists","up"]},{"osd":2,"uuid":"4c89c1b2-aade-402e-b057-44297690cbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":377,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6817","nonce":33227},{"type":"v1","addr":"172.21.15.59:6818","nonce":33227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":33227},{"type":"v1","addr":"172.21.15.59:6823","nonce":33227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":33227},{"type":"v1","addr":"172.21.15.59:6831","nonce":33227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":33227},{"type":"v1","addr":"172.21.15.59:6827","nonce":33227}]},"public_addr":"172.21.15.59:6818/33227","cluster_addr":"172.21.15.59:6823/33227","heartbeat_back_addr":"172.21.15.59:6831/33227","heartbeat_front_addr":"172.21.15.59:6827/33227","state":["exists","up"]},{"osd":3,"uuid":"7b03b655-a408-4331-bc3b-87fc476d7fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":33228},{"type":"v1","addr":"172.21.15.59:6819","nonce":33228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":33228},{"type":"v1","addr":"172.21.15.59:6822","nonce":33228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":33228},{"type":"v1","addr":"172.21.15.59:6830","nonce":33228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":33228},{"type":"v1","addr":"172.21.15.59:6826","nonce":33228}]},"public_addr":"172.21.15.59:6819/33228","cluster_addr":"172.21.15.59:6822/33228","heartbeat_back_addr":"172.21.15.59:6830/33228","heartbeat_front_addr":"172.21.15.59:6826/33228","state":["exists","up"]},{"osd":4,"uuid":"f8d3bc78-4601-40c2-9e29-bc29cb49eb94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":33087},{"type":"v1","addr":"172.21.15.134:6801","nonce":33087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":33087},{"type":"v1","addr":"172.21.15.134:6803","nonce":33087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":33087},{"type":"v1","addr":"172.21.15.134:6807","nonce":33087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":33087},{"type":"v1","addr":"172.21.15.134:6805","nonce":33087}]},"public_addr":"172.21.15.134:6801/33087","cluster_addr":"172.21.15.134:6803/33087","heartbeat_back_addr":"172.21.15.134:6807/33087","heartbeat_front_addr":"172.21.15.134:6805/33087","state":["exists","up"]},{"osd":5,"uuid":"839c9ef0-c471-43b4-8fa4-5f993c2ec671","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":297,"up_from":321,"up_thru":378,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":51924},{"type":"v1","addr":"172.21.15.134:6817","nonce":51924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":51924},{"type":"v1","addr":"172.21.15.134:6819","nonce":51924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6823","nonce":51924},{"type":"v1","addr":"172.21.15.134:6825","nonce":51924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":51924},{"type":"v1","addr":"172.21.15.134:6821","nonce":51924}]},"public_addr":"172.21.15.134:6817/51924","cluster_addr":"172.21.15.134:6819/51924","heartbeat_back_addr":"172.21.15.134:6825/51924","heartbeat_front_addr":"172.21.15.134:6821/51924","state":["exists","up"]},{"osd":6,"uuid":"93af4222-cb52-4fcc-abbe-42c7df688f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":390,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":33088},{"type":"v1","addr":"172.21.15.134:6809","nonce":33088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":33088},{"type":"v1","addr":"172.21.15.134:6811","nonce":33088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":33088},{"type":"v1","addr":"172.21.15.134:6815","nonce":33088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":33088},{"type":"v1","addr":"172.21.15.134:6813","nonce":33088}]},"public_addr":"172.21.15.134:6809/33088","cluster_addr":"172.21.15.134:6811/33088","heartbeat_back_addr":"172.21.15.134:6815/33088","heartbeat_front_addr":"172.21.15.134:6813/33088","state":["exists","up"]},{"osd":7,"uuid":"7325f570-a0fc-468d-8b63-11802b68686c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":33118},{"type":"v1","addr":"172.21.15.134:6824","nonce":33118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":33118},{"type":"v1","addr":"172.21.15.134:6827","nonce":33118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":33118},{"type":"v1","addr":"172.21.15.134:6831","nonce":33118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":33118},{"type":"v1","addr":"172.21.15.134:6829","nonce":33118}]},"public_addr":"172.21.15.134:6824/33118","cluster_addr":"172.21.15.134:6827/33118","heartbeat_back_addr":"172.21.15.134:6831/33118","heartbeat_front_addr":"172.21.15.134:6829/33118","state":["exists","up"]},{"osd":8,"uuid":"e59c0b86-90a7-4fbf-b8e5-bb718050f86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":33587},{"type":"v1","addr":"172.21.15.141:6801","nonce":33587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":33587},{"type":"v1","addr":"172.21.15.141:6803","nonce":33587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":33587},{"type":"v1","addr":"172.21.15.141:6807","nonce":33587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":33587},{"type":"v1","addr":"172.21.15.141:6805","nonce":33587}]},"public_addr":"172.21.15.141:6801/33587","cluster_addr":"172.21.15.141:6803/33587","heartbeat_back_addr":"172.21.15.141:6807/33587","heartbeat_front_addr":"172.21.15.141:6805/33587","state":["exists","up"]},{"osd":9,"uuid":"82a3016c-d9a2-4486-90de-e79a3011c56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":373,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6809","nonce":33588},{"type":"v1","addr":"172.21.15.141:6811","nonce":33588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6813","nonce":33588},{"type":"v1","addr":"172.21.15.141:6815","nonce":33588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6821","nonce":33588},{"type":"v1","addr":"172.21.15.141:6823","nonce":33588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6817","nonce":33588},{"type":"v1","addr":"172.21.15.141:6819","nonce":33588}]},"public_addr":"172.21.15.141:6811/33588","cluster_addr":"172.21.15.141:6815/33588","heartbeat_back_addr":"172.21.15.141:6823/33588","heartbeat_front_addr":"172.21.15.141:6819/33588","state":["exists","up"]},{"osd":10,"uuid":"d24ccde9-6e88-4fd6-b95f-0682b0864c0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":33589},{"type":"v1","addr":"172.21.15.141:6810","nonce":33589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":33589},{"type":"v1","addr":"172.21.15.141:6814","nonce":33589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":33589},{"type":"v1","addr":"172.21.15.141:6822","nonce":33589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":33589},{"type":"v1","addr":"172.21.15.141:6818","nonce":33589}]},"public_addr":"172.21.15.141:6810/33589","cluster_addr":"172.21.15.141:6814/33589","heartbeat_back_addr":"172.21.15.141:6822/33589","heartbeat_front_addr":"172.21.15.141:6818/33589","state":["exists","up"]},{"osd":11,"uuid":"da7db835-7c2d-4329-8f98-5a9164fa3450","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":33590},{"type":"v1","addr":"172.21.15.141:6825","nonce":33590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":33590},{"type":"v1","addr":"172.21.15.141:6827","nonce":33590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":33590},{"type":"v1","addr":"172.21.15.141:6831","nonce":33590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":33590},{"type":"v1","addr":"172.21.15.141:6829","nonce":33590}]},"public_addr":"172.21.15.141:6825/33590","cluster_addr":"172.21.15.141:6827/33590","heartbeat_back_addr":"172.21.15.141:6831/33590","heartbeat_front_addr":"172.21.15.141:6829/33590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:01.845974+0000","dead_epoch":34},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.121271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.669451+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.354991+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.093103+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:19.770700+0000","dead_epoch":298},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.594108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.799685+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.941830+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.824769+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.002341+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.753674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[3,4]},{"pgid":"2.b","osds":[10,11]},{"pgid":"3.c","osds":[11,10,2,3,4,6]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"jerasure21profile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"4","m":"2","name":"jerasure21profile","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-03-07T14:03:43.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats , 2021-03-07T14:03:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:545460846650 2021-03-08T01:48:38.810 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:53.880 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2021-03-07T14:03:53.881 INFO:teuthology.orchestra.run.smithi059.stdout:} , 2021-03-07T14:03:53.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2021-03-08T01:48:38.811 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2021-03-07T14:03:41.804607+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=41, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2021-03-07T14:03:48.250901+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=48, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.044 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2021-03-07T14:02:50.751738+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=50, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) 2021-03-08T01:48:38.812 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:04:40.867 DEBUG:tasks.ceph.mon.c:waiting for process to exit , 2021-03-07T14:04:40.867 INFO:teuthology.orchestra.run:waiting for 300 , 2021-03-07T14:04:40.877 INFO:tasks.ceph.mon.c:Stopped 2021-03-08T01:48:38.813 WARNING:scrape:Ignoring malparsed backtrace: proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() 2021-03-08T01:48:38.813 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2021-03-08T01:48:39.107 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1695-gd5f415b9 (d5f415b9b67a6d66ef39db91322ef188bb9f833e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f3f7e117fc8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2761e2) [0x7f3f7e1181e2] 2021-03-08T01:48:39.109 WARNING:scrape:Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/git.ceph.com_ceph-c_d5f415b9b67a6d66ef39db91322ef188bb9f833e/qa/tasks/rados.py", line 265, in thread 2021-03-08T01:48:39.111 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:43.560 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":404,"fsid":"43481236-7a4d-436b-86b0-491b76d17c89","created":"2021-03-07T13:56:48.736434+0000","modified":"2021-03-07T14:03:42.872342+0000","last_up_change":"2021-03-07T14:02:19.639180+0000","last_in_change":"2021-03-07T14:03:13.676195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-07T13:56:59.279245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"358","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"149","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-07T13:57:01.712160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"111","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-07T13:57:16.685229+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":6,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"372","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":323,"snap_epoch":372,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"jerasure21profile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"b3ff6f99-f475-4496-9836-f0813e4412dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":33,"up_from":127,"up_thru":379,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":51471},{"type":"v1","addr":"172.21.15.59:6801","nonce":51471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":51471},{"type":"v1","addr":"172.21.15.59:6803","nonce":51471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":51471},{"type":"v1","addr":"172.21.15.59:6810","nonce":51471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":51471},{"type":"v1","addr":"172.21.15.59:6806","nonce":51471}]},"public_addr":"172.21.15.59:6801/51471","cluster_addr":"172.21.15.59:6803/51471","heartbeat_back_addr":"172.21.15.59:6810/51471","heartbeat_front_addr":"172.21.15.59:6806/51471","state":["exists","up"]},{"osd":1,"uuid":"7c282045-3f47-4c2f-94bb-1f0be7efd0d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":33226},{"type":"v1","addr":"172.21.15.59:6807","nonce":33226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":33226},{"type":"v1","addr":"172.21.15.59:6811","nonce":33226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":33226},{"type":"v1","addr":"172.21.15.59:6815","nonce":33226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":33226},{"type":"v1","addr":"172.21.15.59:6813","nonce":33226}]},"public_addr":"172.21.15.59:6807/33226","cluster_addr":"172.21.15.59:6811/33226","heartbeat_back_addr":"172.21.15.59:6815/33226","heartbeat_front_addr":"172.21.15.59:6813/33226","state":["exists","up"]},{"osd":2,"uuid":"4c89c1b2-aade-402e-b057-44297690cbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":377,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6817","nonce":33227},{"type":"v1","addr":"172.21.15.59:6818","nonce":33227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":33227},{"type":"v1","addr":"172.21.15.59:6823","nonce":33227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6829","nonce":33227},{"type":"v1","addr":"172.21.15.59:6831","nonce":33227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6825","nonce":33227},{"type":"v1","addr":"172.21.15.59:6827","nonce":33227}]},"public_addr":"172.21.15.59:6818/33227","cluster_addr":"172.21.15.59:6823/33227","heartbeat_back_addr":"172.21.15.59:6831/33227","heartbeat_front_addr":"172.21.15.59:6827/33227","state":["exists","up"]},{"osd":3,"uuid":"7b03b655-a408-4331-bc3b-87fc476d7fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":33228},{"type":"v1","addr":"172.21.15.59:6819","nonce":33228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6821","nonce":33228},{"type":"v1","addr":"172.21.15.59:6822","nonce":33228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":33228},{"type":"v1","addr":"172.21.15.59:6830","nonce":33228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":33228},{"type":"v1","addr":"172.21.15.59:6826","nonce":33228}]},"public_addr":"172.21.15.59:6819/33228","cluster_addr":"172.21.15.59:6822/33228","heartbeat_back_addr":"172.21.15.59:6830/33228","heartbeat_front_addr":"172.21.15.59:6826/33228","state":["exists","up"]},{"osd":4,"uuid":"f8d3bc78-4601-40c2-9e29-bc29cb49eb94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":33087},{"type":"v1","addr":"172.21.15.134:6801","nonce":33087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":33087},{"type":"v1","addr":"172.21.15.134:6803","nonce":33087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":33087},{"type":"v1","addr":"172.21.15.134:6807","nonce":33087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":33087},{"type":"v1","addr":"172.21.15.134:6805","nonce":33087}]},"public_addr":"172.21.15.134:6801/33087","cluster_addr":"172.21.15.134:6803/33087","heartbeat_back_addr":"172.21.15.134:6807/33087","heartbeat_front_addr":"172.21.15.134:6805/33087","state":["exists","up"]},{"osd":5,"uuid":"839c9ef0-c471-43b4-8fa4-5f993c2ec671","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":297,"up_from":321,"up_thru":378,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":51924},{"type":"v1","addr":"172.21.15.134:6817","nonce":51924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":51924},{"type":"v1","addr":"172.21.15.134:6819","nonce":51924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6823","nonce":51924},{"type":"v1","addr":"172.21.15.134:6825","nonce":51924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":51924},{"type":"v1","addr":"172.21.15.134:6821","nonce":51924}]},"public_addr":"172.21.15.134:6817/51924","cluster_addr":"172.21.15.134:6819/51924","heartbeat_back_addr":"172.21.15.134:6825/51924","heartbeat_front_addr":"172.21.15.134:6821/51924","state":["exists","up"]},{"osd":6,"uuid":"93af4222-cb52-4fcc-abbe-42c7df688f59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":390,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":33088},{"type":"v1","addr":"172.21.15.134:6809","nonce":33088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":33088},{"type":"v1","addr":"172.21.15.134:6811","nonce":33088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":33088},{"type":"v1","addr":"172.21.15.134:6815","nonce":33088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":33088},{"type":"v1","addr":"172.21.15.134:6813","nonce":33088}]},"public_addr":"172.21.15.134:6809/33088","cluster_addr":"172.21.15.134:6811/33088","heartbeat_back_addr":"172.21.15.134:6815/33088","heartbeat_front_addr":"172.21.15.134:6813/33088","state":["exists","up"]},{"osd":7,"uuid":"7325f570-a0fc-468d-8b63-11802b68686c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":33118},{"type":"v1","addr":"172.21.15.134:6824","nonce":33118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":33118},{"type":"v1","addr":"172.21.15.134:6827","nonce":33118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":33118},{"type":"v1","addr":"172.21.15.134:6831","nonce":33118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":33118},{"type":"v1","addr":"172.21.15.134:6829","nonce":33118}]},"public_addr":"172.21.15.134:6824/33118","cluster_addr":"172.21.15.134:6827/33118","heartbeat_back_addr":"172.21.15.134:6831/33118","heartbeat_front_addr":"172.21.15.134:6829/33118","state":["exists","up"]},{"osd":8,"uuid":"e59c0b86-90a7-4fbf-b8e5-bb718050f86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":33587},{"type":"v1","addr":"172.21.15.141:6801","nonce":33587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":33587},{"type":"v1","addr":"172.21.15.141:6803","nonce":33587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":33587},{"type":"v1","addr":"172.21.15.141:6807","nonce":33587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":33587},{"type":"v1","addr":"172.21.15.141:6805","nonce":33587}]},"public_addr":"172.21.15.141:6801/33587","cluster_addr":"172.21.15.141:6803/33587","heartbeat_back_addr":"172.21.15.141:6807/33587","heartbeat_front_addr":"172.21.15.141:6805/33587","state":["exists","up"]},{"osd":9,"uuid":"82a3016c-d9a2-4486-90de-e79a3011c56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":373,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6809","nonce":33588},{"type":"v1","addr":"172.21.15.141:6811","nonce":33588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6813","nonce":33588},{"type":"v1","addr":"172.21.15.141:6815","nonce":33588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6821","nonce":33588},{"type":"v1","addr":"172.21.15.141:6823","nonce":33588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6817","nonce":33588},{"type":"v1","addr":"172.21.15.141:6819","nonce":33588}]},"public_addr":"172.21.15.141:6811/33588","cluster_addr":"172.21.15.141:6815/33588","heartbeat_back_addr":"172.21.15.141:6823/33588","heartbeat_front_addr":"172.21.15.141:6819/33588","state":["exists","up"]},{"osd":10,"uuid":"d24ccde9-6e88-4fd6-b95f-0682b0864c0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":378,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":33589},{"type":"v1","addr":"172.21.15.141:6810","nonce":33589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":33589},{"type":"v1","addr":"172.21.15.141:6814","nonce":33589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":33589},{"type":"v1","addr":"172.21.15.141:6822","nonce":33589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":33589},{"type":"v1","addr":"172.21.15.141:6818","nonce":33589}]},"public_addr":"172.21.15.141:6810/33589","cluster_addr":"172.21.15.141:6814/33589","heartbeat_back_addr":"172.21.15.141:6822/33589","heartbeat_front_addr":"172.21.15.141:6818/33589","state":["exists","up"]},{"osd":11,"uuid":"da7db835-7c2d-4329-8f98-5a9164fa3450","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":379,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":33590},{"type":"v1","addr":"172.21.15.141:6825","nonce":33590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":33590},{"type":"v1","addr":"172.21.15.141:6827","nonce":33590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":33590},{"type":"v1","addr":"172.21.15.141:6831","nonce":33590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":33590},{"type":"v1","addr":"172.21.15.141:6829","nonce":33590}]},"public_addr":"172.21.15.141:6825/33590","cluster_addr":"172.21.15.141:6827/33590","heartbeat_back_addr":"172.21.15.141:6831/33590","heartbeat_front_addr":"172.21.15.141:6829/33590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:01.845974+0000","dead_epoch":34},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.121271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.669451+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.354991+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.093103+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:19.770700+0000","dead_epoch":298},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.594108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.799685+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.941830+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.824769+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:03:00.002341+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-07T14:02:59.753674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[3,4]},{"pgid":"2.b","osds":[10,11]},{"pgid":"3.c","osds":[11,10,2,3,4,6]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"jerasure21profile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"4","m":"2","name":"jerasure21profile","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-03-07T14:03:43.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats , 2021-03-07T14:03:43.732 INFO:teuthology.orchestra.run.smithi059.stdout:545460846650 2021-03-08T01:48:39.113 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:53.880 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2021-03-07T14:03:53.881 INFO:teuthology.orchestra.run.smithi059.stdout:} , 2021-03-07T14:03:53.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2021-03-08T01:48:39.114 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2021-03-07T14:03:41.804607+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=41, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.043 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2021-03-07T14:03:48.250901+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=48, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:03:59.044 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2021-03-07T14:02:50.751738+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=50, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=3, tm_sec=51, tm_wday=6, tm_yday=66, tm_isdst=0) 2021-03-08T01:48:39.115 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:04:40.867 DEBUG:tasks.ceph.mon.c:waiting for process to exit , 2021-03-07T14:04:40.867 INFO:teuthology.orchestra.run:waiting for 300 , 2021-03-07T14:04:40.877 INFO:tasks.ceph.mon.c:Stopped 2021-03-08T01:48:39.116 WARNING:scrape:Ignoring malparsed backtrace: proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() 2021-03-08T01:48:39.116 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2021-03-08T01:48:40.443 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1695-gd5f415b9 (d5f415b9b67a6d66ef39db91322ef188bb9f833e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5daff7f29d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5daff7f478] 2021-03-08T01:48:40.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:01:08.397 DEBUG:teuthology.orchestra.run.smithi085:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops , 2021-03-07T14:01:08.441+0000 7f14d2e92700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 17813) UID: 0 , 2021-03-07T14:01:08.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2021-03-08T01:48:40.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:01:09.455 INFO:teuthology.orchestra.run.smithi083.stdout:overload_utilization 0.0047 , 2021-03-07T14:01:09.455 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 weight 0.5486 -> 0.4986 , 2021-03-07T14:01:09.456 INFO:teuthology.orchestra.run.smithi083.stdout:osd.8 weight 0.5785 -> 0.5285 2021-03-08T01:48:40.448 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.10 to 1 (10000) , 2021-03-07T14:01:38.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 , set osd.10 primary-affinity to 1 (8655362) 2021-03-08T01:48:40.538 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:01:54.452+0000 7f25330cc700 -1 osd.3 pg_epoch: 633 pg[3.13s0( v 591'579 (0'0,591'579] local-lis/les=575/576 n=3 ec=508/26 lis/c=575/575 les/c/f=576/576/0 sis=575) [3,0,2,8,1,11]p3(0) r=0 lpr=575 crt=591'579 lcod 590'577 mlcod 590'577 active+clean ps=[1be~1,1d7~1,1e0~1,1e6~1,1e8~1,1ec~2,1f0~1,1f7~1,1f9~1,202~1,206~1,208~1]] scrubber pg(3.13s0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2(2) (0x56483ba0d4a0) , 2021-03-07T14:01:54.452+0000 7f25370d4700 -1 osd.3 pg_epoch: 633 pg[3.13s0( v 591'579 (0'0,591'579] local-lis/les=575/576 n=3 ec=508/26 lis/c=575/575 les/c/f=576/576/0 sis=575) [3,0,2,8,1,11]p3(0) r=0 lpr=575 crt=591'579 lcod 590'577 mlcod 590'577 active+clean ps=[1be~1,1d7~1,1e0~1,1e6~1,1e8~1,1ec~2,1f0~1,1f7~1,1f9~1,202~1,206~1,208~1]] scrubber pg(3.13s0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 11(5) (0x56483ba0e520) , 2021-03-07T14:01:54.643 INFO:teuthology.orchestra.run.smithi083.stdout: 2021-03-08T01:48:40.540 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:02:33.401 INFO:tasks.ceph:Scrubbing osd.4 , 2021-03-07T14:02:33.402 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 , 2021-03-07T14:02:33.582 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2021-03-08T01:48:40.542 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:02:37.848 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-03-07T14:02:01.964156+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=1, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:02:37.849 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2021-03-07T14:02:22.988464+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=22, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:02:37.849 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2021-03-07T14:01:57.525358+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=1, tm_sec=57, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=66, tm_isdst=0) 2021-03-08T01:48:40.543 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:25.655 INFO:tasks.ceph.osd.6:Stopped , 2021-03-07T14:03:25.656 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-03-07T14:03:25.656 INFO:teuthology.orchestra.run:waiting for 300 2021-03-08T01:48:40.543 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:36.851 DEBUG:teuthology.orchestra.run.smithi085:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- , 2021-03-07T14:03:36.874 DEBUG:teuthology.orchestra.run.smithi193:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- , 2021-03-07T14:04:41.541 INFO:tasks.ceph:Archiving logs... 2021-03-08T01:48:40.544 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:05:20.625 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump , 2021-03-07T14:05:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-03-07T14:05:20.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-03-08T01:48:41.022 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1695-gd5f415b9 (d5f415b9b67a6d66ef39db91322ef188bb9f833e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5daff7f29d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5daff7f478] 2021-03-08T01:48:41.023 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:01:08.397 DEBUG:teuthology.orchestra.run.smithi085:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops , 2021-03-07T14:01:08.441+0000 7f14d2e92700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 17813) UID: 0 , 2021-03-07T14:01:08.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2021-03-08T01:48:41.024 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:01:09.455 INFO:teuthology.orchestra.run.smithi083.stdout:overload_utilization 0.0047 , 2021-03-07T14:01:09.455 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 weight 0.5486 -> 0.4986 , 2021-03-07T14:01:09.456 INFO:teuthology.orchestra.run.smithi083.stdout:osd.8 weight 0.5785 -> 0.5285 2021-03-08T01:48:41.027 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.10 to 1 (10000) , 2021-03-07T14:01:38.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 , set osd.10 primary-affinity to 1 (8655362) 2021-03-08T01:48:41.034 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:01:54.452+0000 7f25330cc700 -1 osd.3 pg_epoch: 633 pg[3.13s0( v 591'579 (0'0,591'579] local-lis/les=575/576 n=3 ec=508/26 lis/c=575/575 les/c/f=576/576/0 sis=575) [3,0,2,8,1,11]p3(0) r=0 lpr=575 crt=591'579 lcod 590'577 mlcod 590'577 active+clean ps=[1be~1,1d7~1,1e0~1,1e6~1,1e8~1,1ec~2,1f0~1,1f7~1,1f9~1,202~1,206~1,208~1]] scrubber pg(3.13s0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2(2) (0x56483ba0d4a0) , 2021-03-07T14:01:54.452+0000 7f25370d4700 -1 osd.3 pg_epoch: 633 pg[3.13s0( v 591'579 (0'0,591'579] local-lis/les=575/576 n=3 ec=508/26 lis/c=575/575 les/c/f=576/576/0 sis=575) [3,0,2,8,1,11]p3(0) r=0 lpr=575 crt=591'579 lcod 590'577 mlcod 590'577 active+clean ps=[1be~1,1d7~1,1e0~1,1e6~1,1e8~1,1ec~2,1f0~1,1f7~1,1f9~1,202~1,206~1,208~1]] scrubber pg(3.13s0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 11(5) (0x56483ba0e520) , 2021-03-07T14:01:54.643 INFO:teuthology.orchestra.run.smithi083.stdout: 2021-03-08T01:48:41.036 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:02:33.401 INFO:tasks.ceph:Scrubbing osd.4 , 2021-03-07T14:02:33.402 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 , 2021-03-07T14:02:33.582 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2021-03-08T01:48:41.038 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:02:37.848 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-03-07T14:02:01.964156+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=1, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:02:37.849 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2021-03-07T14:02:22.988464+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=22, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:02:37.849 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2021-03-07T14:01:57.525358+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=1, tm_sec=57, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=66, tm_isdst=0) 2021-03-08T01:48:41.038 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:25.655 INFO:tasks.ceph.osd.6:Stopped , 2021-03-07T14:03:25.656 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-03-07T14:03:25.656 INFO:teuthology.orchestra.run:waiting for 300 2021-03-08T01:48:41.039 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:36.851 DEBUG:teuthology.orchestra.run.smithi085:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- , 2021-03-07T14:03:36.874 DEBUG:teuthology.orchestra.run.smithi193:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- , 2021-03-07T14:04:41.541 INFO:tasks.ceph:Archiving logs... 2021-03-08T01:48:41.040 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:05:20.625 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump , 2021-03-07T14:05:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-03-07T14:05:20.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-03-08T01:48:41.616 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1695-gd5f415b9 (d5f415b9b67a6d66ef39db91322ef188bb9f833e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5daff7f29d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5daff7f478] 2021-03-08T01:48:41.617 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:01:08.397 DEBUG:teuthology.orchestra.run.smithi085:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops , 2021-03-07T14:01:08.441+0000 7f14d2e92700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 17813) UID: 0 , 2021-03-07T14:01:08.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2021-03-08T01:48:41.618 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:01:09.455 INFO:teuthology.orchestra.run.smithi083.stdout:overload_utilization 0.0047 , 2021-03-07T14:01:09.455 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 weight 0.5486 -> 0.4986 , 2021-03-07T14:01:09.456 INFO:teuthology.orchestra.run.smithi083.stdout:osd.8 weight 0.5785 -> 0.5285 2021-03-08T01:48:41.621 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.10 to 1 (10000) , 2021-03-07T14:01:38.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 , set osd.10 primary-affinity to 1 (8655362) 2021-03-08T01:48:41.629 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:01:54.452+0000 7f25330cc700 -1 osd.3 pg_epoch: 633 pg[3.13s0( v 591'579 (0'0,591'579] local-lis/les=575/576 n=3 ec=508/26 lis/c=575/575 les/c/f=576/576/0 sis=575) [3,0,2,8,1,11]p3(0) r=0 lpr=575 crt=591'579 lcod 590'577 mlcod 590'577 active+clean ps=[1be~1,1d7~1,1e0~1,1e6~1,1e8~1,1ec~2,1f0~1,1f7~1,1f9~1,202~1,206~1,208~1]] scrubber pg(3.13s0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2(2) (0x56483ba0d4a0) , 2021-03-07T14:01:54.452+0000 7f25370d4700 -1 osd.3 pg_epoch: 633 pg[3.13s0( v 591'579 (0'0,591'579] local-lis/les=575/576 n=3 ec=508/26 lis/c=575/575 les/c/f=576/576/0 sis=575) [3,0,2,8,1,11]p3(0) r=0 lpr=575 crt=591'579 lcod 590'577 mlcod 590'577 active+clean ps=[1be~1,1d7~1,1e0~1,1e6~1,1e8~1,1ec~2,1f0~1,1f7~1,1f9~1,202~1,206~1,208~1]] scrubber pg(3.13s0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 11(5) (0x56483ba0e520) , 2021-03-07T14:01:54.643 INFO:teuthology.orchestra.run.smithi083.stdout: 2021-03-08T01:48:41.630 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:02:33.401 INFO:tasks.ceph:Scrubbing osd.4 , 2021-03-07T14:02:33.402 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 , 2021-03-07T14:02:33.582 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2021-03-08T01:48:41.632 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:02:37.848 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-03-07T14:02:01.964156+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=1, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:02:37.849 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2021-03-07T14:02:22.988464+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=22, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=66, tm_isdst=0) , 2021-03-07T14:02:37.849 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2021-03-07T14:01:57.525358+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=1, tm_sec=57, tm_wday=6, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=7, tm_hour=14, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=66, tm_isdst=0) 2021-03-08T01:48:41.632 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:25.655 INFO:tasks.ceph.osd.6:Stopped , 2021-03-07T14:03:25.656 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-03-07T14:03:25.656 INFO:teuthology.orchestra.run:waiting for 300 2021-03-08T01:48:41.633 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:03:36.851 DEBUG:teuthology.orchestra.run.smithi085:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- , 2021-03-07T14:03:36.874 DEBUG:teuthology.orchestra.run.smithi193:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- , 2021-03-07T14:04:41.541 INFO:tasks.ceph:Archiving logs... 2021-03-08T01:48:41.634 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T14:05:20.625 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump , 2021-03-07T14:05:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-03-07T14:05:20.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-03-08T01:48:42.287 INFO:scrape:Found 5 distinct failure reasons 2021-03-08T01:48:42.288 INFO:scrape:Failure: Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi200 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d5f415b9b67a6d66ef39db91322ef188bb9f833e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' 2021-03-08T01:48:42.289 INFO:scrape:4 jobs: ['5944427', '5944414', '5944422', '5944409'] 2021-03-08T01:48:42.289 INFO:scrape:suites intersection: ['task/test_e2e}'] 2021-03-08T01:48:42.290 INFO:scrape:suites union: ['rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable', 'rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing', 'task/test_e2e}'] 2021-03-08T01:48:42.290 INFO:scrape: 2021-03-08T01:48:42.291 INFO:scrape:Dead: b'2021-03-08T01:44:45.387 DEBUG:teuthology.exit:Finished running handlers' 2021-03-08T01:48:42.291 INFO:scrape:1 jobs: ['5944408'] 2021-03-08T01:48:42.291 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/fast', 'mon_election/classic', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-radosbench}'] 2021-03-08T01:48:42.292 INFO:scrape: 2021-03-08T01:48:42.292 INFO:scrape:Failure: Command failed on smithi111 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26b27fee-7f4c-11eb-9075-001a4aab830c -e sha1=d5f415b9b67a6d66ef39db91322ef188bb9f833e -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2021-03-08T01:48:42.293 INFO:scrape:2 jobs: ['5944413', '5944426'] 2021-03-08T01:48:42.293 INFO:scrape:suites intersection: ['1-tasks', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-03-08T01:48:42.294 INFO:scrape:suites union: ['1-tasks', 'distro1$/{ubuntu_20.04_kubic_testing}', 'distro1$/{ubuntu_20.04}', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-03-08T01:48:42.294 INFO:scrape: 2021-03-08T01:48:42.294 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' 2021-03-08T01:48:42.295 INFO:scrape:2 jobs: ['5944411', '5944406'] 2021-03-08T01:48:42.295 INFO:scrape:suites intersection: ['cluster/{12-osds', 'mon_election/connectivity', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] 2021-03-08T01:48:42.296 INFO:scrape:suites union: ['cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] 2021-03-08T01:48:42.296 INFO:scrape: 2021-03-08T01:48:42.297 INFO:scrape:Failure: rgw multisite test failures 2021-03-08T01:48:42.297 INFO:scrape:1 jobs: ['5944407'] 2021-03-08T01:48:42.297 INFO:scrape:suites: ['frontend', 'overrides', 'rados/upgrade/pacific-x/rgw-multisite/{clusters', 'realm', 'tasks', 'upgrade/primary}'] 2021-03-08T01:48:42.298 INFO:scrape: 2021-03-08T01:48:42.300 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/ failed: 9 dead: 1 running: 0 waiting: 0 queued: 0 passed: 16 Fail ================================================================= [5944406] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:20:43 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944406/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944406/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5944407] rados/upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:25:55 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944407/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944407/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6db3952a7bc4575918099da6180c535 rgw multisite test failures [5944409] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing task/test_e2e} ----------------------------------------------------------------- time: 00:13:33 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944409/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944409/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b9f2aca54f5847e29aa1a3e3a0b554cd Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d5f415b9b67a6d66ef39db91322ef188bb9f833e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' [5944411] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:16:45 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944411/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944411/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5944413] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_20.04_kubic_testing} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:17:00 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944413/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b94b5c9b5ef4769a284cc119b62c807 Command failed on smithi111 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 26b27fee-7f4c-11eb-9075-001a4aab830c -e sha1=d5f415b9b67a6d66ef39db91322ef188bb9f833e -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [5944414] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable task/test_e2e} ----------------------------------------------------------------- time: 00:13:25 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944414/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944414/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfece3eec14747d2b9c8eedfa5df4427 Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi200 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d5f415b9b67a6d66ef39db91322ef188bb9f833e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' [5944422] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing task/test_e2e} ----------------------------------------------------------------- time: 00:13:27 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944422/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944422/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4456d49cd9b44c78c793ab77c48566c Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi152 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d5f415b9b67a6d66ef39db91322ef188bb9f833e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' [5944426] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_20.04} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:16:46 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944426/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79091650166343658f079127814c297a Command failed on smithi153 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fcd9b708-7f4d-11eb-9075-001a4aab830c -e sha1=d5f415b9b67a6d66ef39db91322ef188bb9f833e -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [5944427] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable task/test_e2e} ----------------------------------------------------------------- time: 00:13:26 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944427/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944427/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8548ca7c7f3243faaea0f72f1dad1c5e Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi200 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d5f415b9b67a6d66ef39db91322ef188bb9f833e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' Dead ================================================================= [5944408] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944408/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944408/ hit max job timeout Pass ================================================================= [5944403] rados/cephadm/smoke/{distro/ubuntu_20.04_kubic_stable fixed-2 mon_election/classic start} time: 00:16:34 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944403/ [5944404] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:11:47 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944404/ [5944405] rados/cephadm/with-work/{0-distro/ubuntu_20.04_kubic_testing fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:24:56 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944405/ [5944410] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:17:20 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944410/ [5944412] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/crush} time: 00:13:21 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944412/ [5944415] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:48 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944415/ [5944416] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:16:11 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944416/ [5944417] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:24 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944417/ [5944418] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:55:07 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944418/ [5944419] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:18 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944419/ [5944420] rados/cephadm/smoke/{distro/ubuntu_20.04_kubic_stable fixed-2 mon_election/classic start} time: 00:15:48 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944420/ [5944421] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:14 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944421/ [5944423] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:27 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944423/ [5944424] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:12:44 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944424/ [5944425] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:10:53 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944425/ [5944428] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:17:42 info: http://pulpito.ceph.com/kchai-2021-03-07_12:30:50-rados-wip-kefu-testing-2021-03-07-1842-distro-basic-smithi/5944428/