Found 38 jobs
Missing teuthology log /home/teuthworker/archive/kchai-2021-03-11_16:25:48-rados-wip-kefu-testing-2021-03-11-2158-distro-basic-gibba/5956161/teuthology.log
Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc9b9981636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7fc9b9981850]

Ignoring malparsed backtrace: 2021-03-11T17:33:18.583 INFO:teuthology.orchestra.run.gibba037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
, 2021-03-11T17:33:18.590 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight
, 2021-03-11T17:33:18.597 INFO:teuthology.orchestra.run.gibba017.stdout:osd.1: {}

Ignoring malparsed backtrace: 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.5191000000000002e-05
, 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:18.927 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.488966+0000",
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.0315999999999998e-05
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.508886+0000",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 2.1806000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.525921+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.037 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 8.6147e-05
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:19.059 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.649523+0000",
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 0.000118886
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:19.081 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.685208+0000",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 7.2374999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.714711+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.133 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:33:22.454+0000 7f8d2c5d0700 -1 WARNING: all dangerous and experimental features are enabled.
, set pool 1 pgp_num to 41
, 2021-03-11T17:33:23.357 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd

Ignoring malparsed backtrace: 2021-03-11T17:33:41.894 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":612,"fsid":"a21ff8f0-d934-473c-8fc0-01ea08a5489d","created":"2021-03-11T17:23:02.442549+0000","modified":"2021-03-11T17:33:41.423164+0000","last_up_change":"2021-03-11T17:33:14.440601+0000","last_in_change":"2021-03-11T17:33:22.334304+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:23:11.576521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":41,"pg_placement_num":41,"pg_placement_num_target":41,"pg_num_target":41,"pg_num_pending":41,"last_pg_merge_meta":{"source_pgid":"1.15","ready_epoch":238,"last_epoch_started":237,"last_epoch_clean":237,"source_version":"0'0","target_version":"0'0"},"last_change":"594","last_force_op_resend":"0","last_force_op_resend_prenautilus":"238","last_force_op_resend_preluminous":"544","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:23:12.713738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"2.8","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"595","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:23:24.612176+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":20,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":20,"last_pg_merge_meta":{"source_pgid":"3.14","ready_epoch":611,"last_epoch_started":610,"last_epoch_clean":610,"source_version":"590'1760","target_version":"566'1467"},"last_change":"611","last_force_op_resend":"0","last_force_op_resend_prenautilus":"611","last_force_op_resend_preluminous":"609","auid":0,"snap_mode":"selfmanaged","snap_seq":485,"snap_epoch":565,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"3bbe5845-5ddc-463f-bb23-ef8c12669fc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":530,"up_from":585,"up_thru":603,"down_at":531,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6802","nonce":101458},{"type":"v1","addr":"172.21.2.117:6803","nonce":101458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":101458},{"type":"v1","addr":"172.21.2.117:6807","nonce":101458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6813","nonce":101458},{"type":"v1","addr":"172.21.2.117:6815","nonce":101458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6809","nonce":101458},{"type":"v1","addr":"172.21.2.117:6811","nonce":101458}]},"public_addr":"172.21.2.117:6803/101458","cluster_addr":"172.21.2.117:6807/101458","heartbeat_back_addr":"172.21.2.117:6815/101458","heartbeat_front_addr":"172.21.2.117:6811/101458","state":["exists","up"]},{"osd":1,"uuid":"70d47745-6d00-4892-898c-a65f11520e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":611,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6819","nonce":35166},{"type":"v1","addr":"172.21.2.117:6821","nonce":35166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6823","nonce":35166},{"type":"v1","addr":"172.21.2.117:6825","nonce":35166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":35166},{"type":"v1","addr":"172.21.2.117:6831","nonce":35166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6827","nonce":35166},{"type":"v1","addr":"172.21.2.117:6829","nonce":35166}]},"public_addr":"172.21.2.117:6821/35166","cluster_addr":"172.21.2.117:6825/35166","heartbeat_back_addr":"172.21.2.117:6831/35166","heartbeat_front_addr":"172.21.2.117:6829/35166","state":["exists","up"]},{"osd":2,"uuid":"fb0542a4-1506-45ac-85f5-b2840d7ac488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":597,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":35167},{"type":"v1","addr":"172.21.2.117:6800","nonce":35167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":35167},{"type":"v1","addr":"172.21.2.117:6806","nonce":35167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":35167},{"type":"v1","addr":"172.21.2.117:6814","nonce":35167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6808","nonce":35167},{"type":"v1","addr":"172.21.2.117:6810","nonce":35167}]},"public_addr":"172.21.2.117:6800/35167","cluster_addr":"172.21.2.117:6806/35167","heartbeat_back_addr":"172.21.2.117:6814/35167","heartbeat_front_addr":"172.21.2.117:6810/35167","state":["exists","up"]},{"osd":3,"uuid":"fcf3328a-38fd-4049-b916-139a8c915fd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":598,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":35168},{"type":"v1","addr":"172.21.2.117:6817","nonce":35168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":35168},{"type":"v1","addr":"172.21.2.117:6820","nonce":35168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":35168},{"type":"v1","addr":"172.21.2.117:6828","nonce":35168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":35168},{"type":"v1","addr":"172.21.2.117:6824","nonce":35168}]},"public_addr":"172.21.2.117:6817/35168","cluster_addr":"172.21.2.117:6820/35168","heartbeat_back_addr":"172.21.2.117:6828/35168","heartbeat_front_addr":"172.21.2.117:6824/35168","state":["exists","up"]},{"osd":4,"uuid":"8fb97983-57f2-4086-b856-4c8cab782051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":599,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6800","nonce":35114},{"type":"v1","addr":"172.21.2.137:6801","nonce":35114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6802","nonce":35114},{"type":"v1","addr":"172.21.2.137:6803","nonce":35114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6806","nonce":35114},{"type":"v1","addr":"172.21.2.137:6807","nonce":35114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6804","nonce":35114},{"type":"v1","addr":"172.21.2.137:6805","nonce":35114}]},"public_addr":"172.21.2.137:6801/35114","cluster_addr":"172.21.2.137:6803/35114","heartbeat_back_addr":"172.21.2.137:6807/35114","heartbeat_front_addr":"172.21.2.137:6805/35114","state":["exists","up"]},{"osd":5,"uuid":"d635fd2e-4bb8-4965-82a9-e1c3b5db179f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":607,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6808","nonce":35115},{"type":"v1","addr":"172.21.2.137:6809","nonce":35115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6810","nonce":35115},{"type":"v1","addr":"172.21.2.137:6811","nonce":35115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6814","nonce":35115},{"type":"v1","addr":"172.21.2.137:6815","nonce":35115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6812","nonce":35115},{"type":"v1","addr":"172.21.2.137:6813","nonce":35115}]},"public_addr":"172.21.2.137:6809/35115","cluster_addr":"172.21.2.137:6811/35115","heartbeat_back_addr":"172.21.2.137:6815/35115","heartbeat_front_addr":"172.21.2.137:6813/35115","state":["exists","up"]},{"osd":6,"uuid":"427e3526-94f6-4820-8e03-86960993a78f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":297,"up_from":335,"up_thru":607,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6816","nonce":57643},{"type":"v1","addr":"172.21.2.137:6817","nonce":57643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6818","nonce":57643},{"type":"v1","addr":"172.21.2.137:6819","nonce":57643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6823","nonce":57643},{"type":"v1","addr":"172.21.2.137:6825","nonce":57643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6820","nonce":57643},{"type":"v1","addr":"172.21.2.137:6821","nonce":57643}]},"public_addr":"172.21.2.137:6817/57643","cluster_addr":"172.21.2.137:6819/57643","heartbeat_back_addr":"172.21.2.137:6825/57643","heartbeat_front_addr":"172.21.2.137:6821/57643","state":["exists","up"]},{"osd":7,"uuid":"27250a3c-2474-4d7d-a2dd-fc2751e3f853","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":257,"up_from":274,"up_thru":607,"down_at":258,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6822","nonce":53803},{"type":"v1","addr":"172.21.2.137:6824","nonce":53803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6826","nonce":53803},{"type":"v1","addr":"172.21.2.137:6827","nonce":53803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6830","nonce":53803},{"type":"v1","addr":"172.21.2.137:6831","nonce":53803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6828","nonce":53803},{"type":"v1","addr":"172.21.2.137:6829","nonce":53803}]},"public_addr":"172.21.2.137:6824/53803","cluster_addr":"172.21.2.137:6827/53803","heartbeat_back_addr":"172.21.2.137:6831/53803","heartbeat_front_addr":"172.21.2.137:6829/53803","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.187728+0000","dead_epoch":531},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.282348+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.265358+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:13.009966+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.849349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.470760+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.749489+0000","dead_epoch":298},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.855663+0000","dead_epoch":258}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.14","mappings":[{"from":2,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:33:41.905 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats
, 2021-03-11T17:33:41.994+0000 7ffa87322700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:33:48.000 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout
, 2021-03-11T17:33:48.001 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-03-11T17:33:48.087+0000 7eff86df1700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:05.277 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:34:05.364+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.
, 2021-03-11T17:34:05.375+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2021-03-11T17:33:46.301750+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=46, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.12 last_scrub_stamp 2021-03-11T17:33:33.190052+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=33, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.095 INFO:tasks.ceph:pgid 1.13 last_scrub_stamp 2021-03-11T17:33:45.993208+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=45, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:35:09.132 INFO:teuthology.misc:Shutting down mds daemons...
, 2021-03-11T17:35:09.133 INFO:teuthology.misc:Shutting down osd daemons...
, 2021-03-11T17:35:09.133 DEBUG:tasks.ceph.osd.0:waiting for process to exit

Ignoring malparsed backtrace: 2021-03-11T17:36:56.988 DEBUG:teuthology.orchestra.run.gibba017:> sudo tar cz -f - -C /var/log/ceph -- .
, 2021-03-11T17:37:24.809 DEBUG:teuthology.misc:Transferring archived files from gibba037:/var/log/ceph to /home/teuthworker/archive/kchai-2021-03-11_16:25:48-rados-wip-kefu-testing-2021-03-11-2158-distro-basic-gibba/5956183/remote/gibba037/log
, 2021-03-11T17:37:24.814 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- .

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc9b9981636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7fc9b9981850]

Ignoring malparsed backtrace: 2021-03-11T17:33:18.583 INFO:teuthology.orchestra.run.gibba037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
, 2021-03-11T17:33:18.590 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight
, 2021-03-11T17:33:18.597 INFO:teuthology.orchestra.run.gibba017.stdout:osd.1: {}

Ignoring malparsed backtrace: 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.5191000000000002e-05
, 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:18.927 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.488966+0000",
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.0315999999999998e-05
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.508886+0000",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 2.1806000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.525921+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.037 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 8.6147e-05
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:19.059 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.649523+0000",
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 0.000118886
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:19.081 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.685208+0000",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 7.2374999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.714711+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.133 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:33:22.454+0000 7f8d2c5d0700 -1 WARNING: all dangerous and experimental features are enabled.
, set pool 1 pgp_num to 41
, 2021-03-11T17:33:23.357 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd

Ignoring malparsed backtrace: 2021-03-11T17:33:41.894 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":612,"fsid":"a21ff8f0-d934-473c-8fc0-01ea08a5489d","created":"2021-03-11T17:23:02.442549+0000","modified":"2021-03-11T17:33:41.423164+0000","last_up_change":"2021-03-11T17:33:14.440601+0000","last_in_change":"2021-03-11T17:33:22.334304+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:23:11.576521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":41,"pg_placement_num":41,"pg_placement_num_target":41,"pg_num_target":41,"pg_num_pending":41,"last_pg_merge_meta":{"source_pgid":"1.15","ready_epoch":238,"last_epoch_started":237,"last_epoch_clean":237,"source_version":"0'0","target_version":"0'0"},"last_change":"594","last_force_op_resend":"0","last_force_op_resend_prenautilus":"238","last_force_op_resend_preluminous":"544","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:23:12.713738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"2.8","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"595","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:23:24.612176+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":20,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":20,"last_pg_merge_meta":{"source_pgid":"3.14","ready_epoch":611,"last_epoch_started":610,"last_epoch_clean":610,"source_version":"590'1760","target_version":"566'1467"},"last_change":"611","last_force_op_resend":"0","last_force_op_resend_prenautilus":"611","last_force_op_resend_preluminous":"609","auid":0,"snap_mode":"selfmanaged","snap_seq":485,"snap_epoch":565,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"3bbe5845-5ddc-463f-bb23-ef8c12669fc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":530,"up_from":585,"up_thru":603,"down_at":531,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6802","nonce":101458},{"type":"v1","addr":"172.21.2.117:6803","nonce":101458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":101458},{"type":"v1","addr":"172.21.2.117:6807","nonce":101458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6813","nonce":101458},{"type":"v1","addr":"172.21.2.117:6815","nonce":101458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6809","nonce":101458},{"type":"v1","addr":"172.21.2.117:6811","nonce":101458}]},"public_addr":"172.21.2.117:6803/101458","cluster_addr":"172.21.2.117:6807/101458","heartbeat_back_addr":"172.21.2.117:6815/101458","heartbeat_front_addr":"172.21.2.117:6811/101458","state":["exists","up"]},{"osd":1,"uuid":"70d47745-6d00-4892-898c-a65f11520e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":611,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6819","nonce":35166},{"type":"v1","addr":"172.21.2.117:6821","nonce":35166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6823","nonce":35166},{"type":"v1","addr":"172.21.2.117:6825","nonce":35166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":35166},{"type":"v1","addr":"172.21.2.117:6831","nonce":35166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6827","nonce":35166},{"type":"v1","addr":"172.21.2.117:6829","nonce":35166}]},"public_addr":"172.21.2.117:6821/35166","cluster_addr":"172.21.2.117:6825/35166","heartbeat_back_addr":"172.21.2.117:6831/35166","heartbeat_front_addr":"172.21.2.117:6829/35166","state":["exists","up"]},{"osd":2,"uuid":"fb0542a4-1506-45ac-85f5-b2840d7ac488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":597,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":35167},{"type":"v1","addr":"172.21.2.117:6800","nonce":35167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":35167},{"type":"v1","addr":"172.21.2.117:6806","nonce":35167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":35167},{"type":"v1","addr":"172.21.2.117:6814","nonce":35167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6808","nonce":35167},{"type":"v1","addr":"172.21.2.117:6810","nonce":35167}]},"public_addr":"172.21.2.117:6800/35167","cluster_addr":"172.21.2.117:6806/35167","heartbeat_back_addr":"172.21.2.117:6814/35167","heartbeat_front_addr":"172.21.2.117:6810/35167","state":["exists","up"]},{"osd":3,"uuid":"fcf3328a-38fd-4049-b916-139a8c915fd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":598,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":35168},{"type":"v1","addr":"172.21.2.117:6817","nonce":35168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":35168},{"type":"v1","addr":"172.21.2.117:6820","nonce":35168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":35168},{"type":"v1","addr":"172.21.2.117:6828","nonce":35168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":35168},{"type":"v1","addr":"172.21.2.117:6824","nonce":35168}]},"public_addr":"172.21.2.117:6817/35168","cluster_addr":"172.21.2.117:6820/35168","heartbeat_back_addr":"172.21.2.117:6828/35168","heartbeat_front_addr":"172.21.2.117:6824/35168","state":["exists","up"]},{"osd":4,"uuid":"8fb97983-57f2-4086-b856-4c8cab782051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":599,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6800","nonce":35114},{"type":"v1","addr":"172.21.2.137:6801","nonce":35114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6802","nonce":35114},{"type":"v1","addr":"172.21.2.137:6803","nonce":35114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6806","nonce":35114},{"type":"v1","addr":"172.21.2.137:6807","nonce":35114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6804","nonce":35114},{"type":"v1","addr":"172.21.2.137:6805","nonce":35114}]},"public_addr":"172.21.2.137:6801/35114","cluster_addr":"172.21.2.137:6803/35114","heartbeat_back_addr":"172.21.2.137:6807/35114","heartbeat_front_addr":"172.21.2.137:6805/35114","state":["exists","up"]},{"osd":5,"uuid":"d635fd2e-4bb8-4965-82a9-e1c3b5db179f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":607,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6808","nonce":35115},{"type":"v1","addr":"172.21.2.137:6809","nonce":35115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6810","nonce":35115},{"type":"v1","addr":"172.21.2.137:6811","nonce":35115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6814","nonce":35115},{"type":"v1","addr":"172.21.2.137:6815","nonce":35115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6812","nonce":35115},{"type":"v1","addr":"172.21.2.137:6813","nonce":35115}]},"public_addr":"172.21.2.137:6809/35115","cluster_addr":"172.21.2.137:6811/35115","heartbeat_back_addr":"172.21.2.137:6815/35115","heartbeat_front_addr":"172.21.2.137:6813/35115","state":["exists","up"]},{"osd":6,"uuid":"427e3526-94f6-4820-8e03-86960993a78f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":297,"up_from":335,"up_thru":607,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6816","nonce":57643},{"type":"v1","addr":"172.21.2.137:6817","nonce":57643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6818","nonce":57643},{"type":"v1","addr":"172.21.2.137:6819","nonce":57643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6823","nonce":57643},{"type":"v1","addr":"172.21.2.137:6825","nonce":57643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6820","nonce":57643},{"type":"v1","addr":"172.21.2.137:6821","nonce":57643}]},"public_addr":"172.21.2.137:6817/57643","cluster_addr":"172.21.2.137:6819/57643","heartbeat_back_addr":"172.21.2.137:6825/57643","heartbeat_front_addr":"172.21.2.137:6821/57643","state":["exists","up"]},{"osd":7,"uuid":"27250a3c-2474-4d7d-a2dd-fc2751e3f853","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":257,"up_from":274,"up_thru":607,"down_at":258,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6822","nonce":53803},{"type":"v1","addr":"172.21.2.137:6824","nonce":53803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6826","nonce":53803},{"type":"v1","addr":"172.21.2.137:6827","nonce":53803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6830","nonce":53803},{"type":"v1","addr":"172.21.2.137:6831","nonce":53803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6828","nonce":53803},{"type":"v1","addr":"172.21.2.137:6829","nonce":53803}]},"public_addr":"172.21.2.137:6824/53803","cluster_addr":"172.21.2.137:6827/53803","heartbeat_back_addr":"172.21.2.137:6831/53803","heartbeat_front_addr":"172.21.2.137:6829/53803","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.187728+0000","dead_epoch":531},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.282348+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.265358+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:13.009966+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.849349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.470760+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.749489+0000","dead_epoch":298},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.855663+0000","dead_epoch":258}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.14","mappings":[{"from":2,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:33:41.905 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats
, 2021-03-11T17:33:41.994+0000 7ffa87322700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:33:48.000 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout
, 2021-03-11T17:33:48.001 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-03-11T17:33:48.087+0000 7eff86df1700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:05.277 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:34:05.364+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.
, 2021-03-11T17:34:05.375+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2021-03-11T17:33:46.301750+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=46, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.12 last_scrub_stamp 2021-03-11T17:33:33.190052+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=33, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.095 INFO:tasks.ceph:pgid 1.13 last_scrub_stamp 2021-03-11T17:33:45.993208+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=45, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:35:09.132 INFO:teuthology.misc:Shutting down mds daemons...
, 2021-03-11T17:35:09.133 INFO:teuthology.misc:Shutting down osd daemons...
, 2021-03-11T17:35:09.133 DEBUG:tasks.ceph.osd.0:waiting for process to exit

Ignoring malparsed backtrace: 2021-03-11T17:36:56.988 DEBUG:teuthology.orchestra.run.gibba017:> sudo tar cz -f - -C /var/log/ceph -- .
, 2021-03-11T17:37:24.809 DEBUG:teuthology.misc:Transferring archived files from gibba037:/var/log/ceph to /home/teuthworker/archive/kchai-2021-03-11_16:25:48-rados-wip-kefu-testing-2021-03-11-2158-distro-basic-gibba/5956183/remote/gibba037/log
, 2021-03-11T17:37:24.814 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- .

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc9b9981636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7fc9b9981850]

Ignoring malparsed backtrace: 2021-03-11T17:33:18.583 INFO:teuthology.orchestra.run.gibba037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
, 2021-03-11T17:33:18.590 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight
, 2021-03-11T17:33:18.597 INFO:teuthology.orchestra.run.gibba017.stdout:osd.1: {}

Ignoring malparsed backtrace: 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.5191000000000002e-05
, 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:18.927 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.488966+0000",
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.0315999999999998e-05
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.508886+0000",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 2.1806000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.525921+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.037 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 8.6147e-05
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:19.059 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.649523+0000",
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 0.000118886
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:19.081 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.685208+0000",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 7.2374999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.714711+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.133 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:33:22.454+0000 7f8d2c5d0700 -1 WARNING: all dangerous and experimental features are enabled.
, set pool 1 pgp_num to 41
, 2021-03-11T17:33:23.357 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd

Ignoring malparsed backtrace: 2021-03-11T17:33:41.894 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":612,"fsid":"a21ff8f0-d934-473c-8fc0-01ea08a5489d","created":"2021-03-11T17:23:02.442549+0000","modified":"2021-03-11T17:33:41.423164+0000","last_up_change":"2021-03-11T17:33:14.440601+0000","last_in_change":"2021-03-11T17:33:22.334304+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:23:11.576521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":41,"pg_placement_num":41,"pg_placement_num_target":41,"pg_num_target":41,"pg_num_pending":41,"last_pg_merge_meta":{"source_pgid":"1.15","ready_epoch":238,"last_epoch_started":237,"last_epoch_clean":237,"source_version":"0'0","target_version":"0'0"},"last_change":"594","last_force_op_resend":"0","last_force_op_resend_prenautilus":"238","last_force_op_resend_preluminous":"544","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:23:12.713738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"2.8","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"595","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:23:24.612176+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":20,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":20,"last_pg_merge_meta":{"source_pgid":"3.14","ready_epoch":611,"last_epoch_started":610,"last_epoch_clean":610,"source_version":"590'1760","target_version":"566'1467"},"last_change":"611","last_force_op_resend":"0","last_force_op_resend_prenautilus":"611","last_force_op_resend_preluminous":"609","auid":0,"snap_mode":"selfmanaged","snap_seq":485,"snap_epoch":565,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"3bbe5845-5ddc-463f-bb23-ef8c12669fc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":530,"up_from":585,"up_thru":603,"down_at":531,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6802","nonce":101458},{"type":"v1","addr":"172.21.2.117:6803","nonce":101458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":101458},{"type":"v1","addr":"172.21.2.117:6807","nonce":101458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6813","nonce":101458},{"type":"v1","addr":"172.21.2.117:6815","nonce":101458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6809","nonce":101458},{"type":"v1","addr":"172.21.2.117:6811","nonce":101458}]},"public_addr":"172.21.2.117:6803/101458","cluster_addr":"172.21.2.117:6807/101458","heartbeat_back_addr":"172.21.2.117:6815/101458","heartbeat_front_addr":"172.21.2.117:6811/101458","state":["exists","up"]},{"osd":1,"uuid":"70d47745-6d00-4892-898c-a65f11520e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":611,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6819","nonce":35166},{"type":"v1","addr":"172.21.2.117:6821","nonce":35166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6823","nonce":35166},{"type":"v1","addr":"172.21.2.117:6825","nonce":35166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":35166},{"type":"v1","addr":"172.21.2.117:6831","nonce":35166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6827","nonce":35166},{"type":"v1","addr":"172.21.2.117:6829","nonce":35166}]},"public_addr":"172.21.2.117:6821/35166","cluster_addr":"172.21.2.117:6825/35166","heartbeat_back_addr":"172.21.2.117:6831/35166","heartbeat_front_addr":"172.21.2.117:6829/35166","state":["exists","up"]},{"osd":2,"uuid":"fb0542a4-1506-45ac-85f5-b2840d7ac488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":597,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":35167},{"type":"v1","addr":"172.21.2.117:6800","nonce":35167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":35167},{"type":"v1","addr":"172.21.2.117:6806","nonce":35167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":35167},{"type":"v1","addr":"172.21.2.117:6814","nonce":35167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6808","nonce":35167},{"type":"v1","addr":"172.21.2.117:6810","nonce":35167}]},"public_addr":"172.21.2.117:6800/35167","cluster_addr":"172.21.2.117:6806/35167","heartbeat_back_addr":"172.21.2.117:6814/35167","heartbeat_front_addr":"172.21.2.117:6810/35167","state":["exists","up"]},{"osd":3,"uuid":"fcf3328a-38fd-4049-b916-139a8c915fd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":598,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":35168},{"type":"v1","addr":"172.21.2.117:6817","nonce":35168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":35168},{"type":"v1","addr":"172.21.2.117:6820","nonce":35168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":35168},{"type":"v1","addr":"172.21.2.117:6828","nonce":35168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":35168},{"type":"v1","addr":"172.21.2.117:6824","nonce":35168}]},"public_addr":"172.21.2.117:6817/35168","cluster_addr":"172.21.2.117:6820/35168","heartbeat_back_addr":"172.21.2.117:6828/35168","heartbeat_front_addr":"172.21.2.117:6824/35168","state":["exists","up"]},{"osd":4,"uuid":"8fb97983-57f2-4086-b856-4c8cab782051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":599,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6800","nonce":35114},{"type":"v1","addr":"172.21.2.137:6801","nonce":35114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6802","nonce":35114},{"type":"v1","addr":"172.21.2.137:6803","nonce":35114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6806","nonce":35114},{"type":"v1","addr":"172.21.2.137:6807","nonce":35114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6804","nonce":35114},{"type":"v1","addr":"172.21.2.137:6805","nonce":35114}]},"public_addr":"172.21.2.137:6801/35114","cluster_addr":"172.21.2.137:6803/35114","heartbeat_back_addr":"172.21.2.137:6807/35114","heartbeat_front_addr":"172.21.2.137:6805/35114","state":["exists","up"]},{"osd":5,"uuid":"d635fd2e-4bb8-4965-82a9-e1c3b5db179f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":607,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6808","nonce":35115},{"type":"v1","addr":"172.21.2.137:6809","nonce":35115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6810","nonce":35115},{"type":"v1","addr":"172.21.2.137:6811","nonce":35115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6814","nonce":35115},{"type":"v1","addr":"172.21.2.137:6815","nonce":35115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6812","nonce":35115},{"type":"v1","addr":"172.21.2.137:6813","nonce":35115}]},"public_addr":"172.21.2.137:6809/35115","cluster_addr":"172.21.2.137:6811/35115","heartbeat_back_addr":"172.21.2.137:6815/35115","heartbeat_front_addr":"172.21.2.137:6813/35115","state":["exists","up"]},{"osd":6,"uuid":"427e3526-94f6-4820-8e03-86960993a78f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":297,"up_from":335,"up_thru":607,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6816","nonce":57643},{"type":"v1","addr":"172.21.2.137:6817","nonce":57643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6818","nonce":57643},{"type":"v1","addr":"172.21.2.137:6819","nonce":57643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6823","nonce":57643},{"type":"v1","addr":"172.21.2.137:6825","nonce":57643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6820","nonce":57643},{"type":"v1","addr":"172.21.2.137:6821","nonce":57643}]},"public_addr":"172.21.2.137:6817/57643","cluster_addr":"172.21.2.137:6819/57643","heartbeat_back_addr":"172.21.2.137:6825/57643","heartbeat_front_addr":"172.21.2.137:6821/57643","state":["exists","up"]},{"osd":7,"uuid":"27250a3c-2474-4d7d-a2dd-fc2751e3f853","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":257,"up_from":274,"up_thru":607,"down_at":258,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6822","nonce":53803},{"type":"v1","addr":"172.21.2.137:6824","nonce":53803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6826","nonce":53803},{"type":"v1","addr":"172.21.2.137:6827","nonce":53803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6830","nonce":53803},{"type":"v1","addr":"172.21.2.137:6831","nonce":53803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6828","nonce":53803},{"type":"v1","addr":"172.21.2.137:6829","nonce":53803}]},"public_addr":"172.21.2.137:6824/53803","cluster_addr":"172.21.2.137:6827/53803","heartbeat_back_addr":"172.21.2.137:6831/53803","heartbeat_front_addr":"172.21.2.137:6829/53803","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.187728+0000","dead_epoch":531},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.282348+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.265358+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:13.009966+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.849349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.470760+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.749489+0000","dead_epoch":298},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.855663+0000","dead_epoch":258}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.14","mappings":[{"from":2,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:33:41.905 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats
, 2021-03-11T17:33:41.994+0000 7ffa87322700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:33:48.000 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout
, 2021-03-11T17:33:48.001 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-03-11T17:33:48.087+0000 7eff86df1700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:05.277 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:34:05.364+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.
, 2021-03-11T17:34:05.375+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2021-03-11T17:33:46.301750+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=46, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.12 last_scrub_stamp 2021-03-11T17:33:33.190052+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=33, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.095 INFO:tasks.ceph:pgid 1.13 last_scrub_stamp 2021-03-11T17:33:45.993208+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=45, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:35:09.132 INFO:teuthology.misc:Shutting down mds daemons...
, 2021-03-11T17:35:09.133 INFO:teuthology.misc:Shutting down osd daemons...
, 2021-03-11T17:35:09.133 DEBUG:tasks.ceph.osd.0:waiting for process to exit

Ignoring malparsed backtrace: 2021-03-11T17:36:56.988 DEBUG:teuthology.orchestra.run.gibba017:> sudo tar cz -f - -C /var/log/ceph -- .
, 2021-03-11T17:37:24.809 DEBUG:teuthology.misc:Transferring archived files from gibba037:/var/log/ceph to /home/teuthworker/archive/kchai-2021-03-11_16:25:48-rados-wip-kefu-testing-2021-03-11-2158-distro-basic-gibba/5956183/remote/gibba037/log
, 2021-03-11T17:37:24.814 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- .

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc9b9981636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7fc9b9981850]

Ignoring malparsed backtrace: 2021-03-11T17:33:18.583 INFO:teuthology.orchestra.run.gibba037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
, 2021-03-11T17:33:18.590 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight
, 2021-03-11T17:33:18.597 INFO:teuthology.orchestra.run.gibba017.stdout:osd.1: {}

Ignoring malparsed backtrace: 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.5191000000000002e-05
, 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:18.927 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.488966+0000",
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.0315999999999998e-05
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.508886+0000",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 2.1806000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.525921+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.037 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 8.6147e-05
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:19.059 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.649523+0000",
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 0.000118886
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:19.081 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.685208+0000",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 7.2374999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.714711+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.133 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:33:22.454+0000 7f8d2c5d0700 -1 WARNING: all dangerous and experimental features are enabled.
, set pool 1 pgp_num to 41
, 2021-03-11T17:33:23.357 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd

Ignoring malparsed backtrace: 2021-03-11T17:33:41.894 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":612,"fsid":"a21ff8f0-d934-473c-8fc0-01ea08a5489d","created":"2021-03-11T17:23:02.442549+0000","modified":"2021-03-11T17:33:41.423164+0000","last_up_change":"2021-03-11T17:33:14.440601+0000","last_in_change":"2021-03-11T17:33:22.334304+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:23:11.576521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":41,"pg_placement_num":41,"pg_placement_num_target":41,"pg_num_target":41,"pg_num_pending":41,"last_pg_merge_meta":{"source_pgid":"1.15","ready_epoch":238,"last_epoch_started":237,"last_epoch_clean":237,"source_version":"0'0","target_version":"0'0"},"last_change":"594","last_force_op_resend":"0","last_force_op_resend_prenautilus":"238","last_force_op_resend_preluminous":"544","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:23:12.713738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"2.8","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"595","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:23:24.612176+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":20,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":20,"last_pg_merge_meta":{"source_pgid":"3.14","ready_epoch":611,"last_epoch_started":610,"last_epoch_clean":610,"source_version":"590'1760","target_version":"566'1467"},"last_change":"611","last_force_op_resend":"0","last_force_op_resend_prenautilus":"611","last_force_op_resend_preluminous":"609","auid":0,"snap_mode":"selfmanaged","snap_seq":485,"snap_epoch":565,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"3bbe5845-5ddc-463f-bb23-ef8c12669fc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":530,"up_from":585,"up_thru":603,"down_at":531,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6802","nonce":101458},{"type":"v1","addr":"172.21.2.117:6803","nonce":101458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":101458},{"type":"v1","addr":"172.21.2.117:6807","nonce":101458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6813","nonce":101458},{"type":"v1","addr":"172.21.2.117:6815","nonce":101458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6809","nonce":101458},{"type":"v1","addr":"172.21.2.117:6811","nonce":101458}]},"public_addr":"172.21.2.117:6803/101458","cluster_addr":"172.21.2.117:6807/101458","heartbeat_back_addr":"172.21.2.117:6815/101458","heartbeat_front_addr":"172.21.2.117:6811/101458","state":["exists","up"]},{"osd":1,"uuid":"70d47745-6d00-4892-898c-a65f11520e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":611,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6819","nonce":35166},{"type":"v1","addr":"172.21.2.117:6821","nonce":35166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6823","nonce":35166},{"type":"v1","addr":"172.21.2.117:6825","nonce":35166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":35166},{"type":"v1","addr":"172.21.2.117:6831","nonce":35166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6827","nonce":35166},{"type":"v1","addr":"172.21.2.117:6829","nonce":35166}]},"public_addr":"172.21.2.117:6821/35166","cluster_addr":"172.21.2.117:6825/35166","heartbeat_back_addr":"172.21.2.117:6831/35166","heartbeat_front_addr":"172.21.2.117:6829/35166","state":["exists","up"]},{"osd":2,"uuid":"fb0542a4-1506-45ac-85f5-b2840d7ac488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":597,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":35167},{"type":"v1","addr":"172.21.2.117:6800","nonce":35167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":35167},{"type":"v1","addr":"172.21.2.117:6806","nonce":35167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":35167},{"type":"v1","addr":"172.21.2.117:6814","nonce":35167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6808","nonce":35167},{"type":"v1","addr":"172.21.2.117:6810","nonce":35167}]},"public_addr":"172.21.2.117:6800/35167","cluster_addr":"172.21.2.117:6806/35167","heartbeat_back_addr":"172.21.2.117:6814/35167","heartbeat_front_addr":"172.21.2.117:6810/35167","state":["exists","up"]},{"osd":3,"uuid":"fcf3328a-38fd-4049-b916-139a8c915fd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":598,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":35168},{"type":"v1","addr":"172.21.2.117:6817","nonce":35168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":35168},{"type":"v1","addr":"172.21.2.117:6820","nonce":35168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":35168},{"type":"v1","addr":"172.21.2.117:6828","nonce":35168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":35168},{"type":"v1","addr":"172.21.2.117:6824","nonce":35168}]},"public_addr":"172.21.2.117:6817/35168","cluster_addr":"172.21.2.117:6820/35168","heartbeat_back_addr":"172.21.2.117:6828/35168","heartbeat_front_addr":"172.21.2.117:6824/35168","state":["exists","up"]},{"osd":4,"uuid":"8fb97983-57f2-4086-b856-4c8cab782051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":599,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6800","nonce":35114},{"type":"v1","addr":"172.21.2.137:6801","nonce":35114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6802","nonce":35114},{"type":"v1","addr":"172.21.2.137:6803","nonce":35114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6806","nonce":35114},{"type":"v1","addr":"172.21.2.137:6807","nonce":35114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6804","nonce":35114},{"type":"v1","addr":"172.21.2.137:6805","nonce":35114}]},"public_addr":"172.21.2.137:6801/35114","cluster_addr":"172.21.2.137:6803/35114","heartbeat_back_addr":"172.21.2.137:6807/35114","heartbeat_front_addr":"172.21.2.137:6805/35114","state":["exists","up"]},{"osd":5,"uuid":"d635fd2e-4bb8-4965-82a9-e1c3b5db179f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":607,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6808","nonce":35115},{"type":"v1","addr":"172.21.2.137:6809","nonce":35115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6810","nonce":35115},{"type":"v1","addr":"172.21.2.137:6811","nonce":35115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6814","nonce":35115},{"type":"v1","addr":"172.21.2.137:6815","nonce":35115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6812","nonce":35115},{"type":"v1","addr":"172.21.2.137:6813","nonce":35115}]},"public_addr":"172.21.2.137:6809/35115","cluster_addr":"172.21.2.137:6811/35115","heartbeat_back_addr":"172.21.2.137:6815/35115","heartbeat_front_addr":"172.21.2.137:6813/35115","state":["exists","up"]},{"osd":6,"uuid":"427e3526-94f6-4820-8e03-86960993a78f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":297,"up_from":335,"up_thru":607,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6816","nonce":57643},{"type":"v1","addr":"172.21.2.137:6817","nonce":57643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6818","nonce":57643},{"type":"v1","addr":"172.21.2.137:6819","nonce":57643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6823","nonce":57643},{"type":"v1","addr":"172.21.2.137:6825","nonce":57643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6820","nonce":57643},{"type":"v1","addr":"172.21.2.137:6821","nonce":57643}]},"public_addr":"172.21.2.137:6817/57643","cluster_addr":"172.21.2.137:6819/57643","heartbeat_back_addr":"172.21.2.137:6825/57643","heartbeat_front_addr":"172.21.2.137:6821/57643","state":["exists","up"]},{"osd":7,"uuid":"27250a3c-2474-4d7d-a2dd-fc2751e3f853","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":257,"up_from":274,"up_thru":607,"down_at":258,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6822","nonce":53803},{"type":"v1","addr":"172.21.2.137:6824","nonce":53803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6826","nonce":53803},{"type":"v1","addr":"172.21.2.137:6827","nonce":53803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6830","nonce":53803},{"type":"v1","addr":"172.21.2.137:6831","nonce":53803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6828","nonce":53803},{"type":"v1","addr":"172.21.2.137:6829","nonce":53803}]},"public_addr":"172.21.2.137:6824/53803","cluster_addr":"172.21.2.137:6827/53803","heartbeat_back_addr":"172.21.2.137:6831/53803","heartbeat_front_addr":"172.21.2.137:6829/53803","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.187728+0000","dead_epoch":531},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.282348+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.265358+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:13.009966+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.849349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.470760+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.749489+0000","dead_epoch":298},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.855663+0000","dead_epoch":258}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.14","mappings":[{"from":2,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:33:41.905 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats
, 2021-03-11T17:33:41.994+0000 7ffa87322700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:33:48.000 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout
, 2021-03-11T17:33:48.001 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-03-11T17:33:48.087+0000 7eff86df1700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:05.277 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:34:05.364+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.
, 2021-03-11T17:34:05.375+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2021-03-11T17:33:46.301750+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=46, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.12 last_scrub_stamp 2021-03-11T17:33:33.190052+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=33, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.095 INFO:tasks.ceph:pgid 1.13 last_scrub_stamp 2021-03-11T17:33:45.993208+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=45, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:35:09.132 INFO:teuthology.misc:Shutting down mds daemons...
, 2021-03-11T17:35:09.133 INFO:teuthology.misc:Shutting down osd daemons...
, 2021-03-11T17:35:09.133 DEBUG:tasks.ceph.osd.0:waiting for process to exit

Ignoring malparsed backtrace: 2021-03-11T17:36:56.988 DEBUG:teuthology.orchestra.run.gibba017:> sudo tar cz -f - -C /var/log/ceph -- .
, 2021-03-11T17:37:24.809 DEBUG:teuthology.misc:Transferring archived files from gibba037:/var/log/ceph to /home/teuthworker/archive/kchai-2021-03-11_16:25:48-rados-wip-kefu-testing-2021-03-11-2158-distro-basic-gibba/5956183/remote/gibba037/log
, 2021-03-11T17:37:24.814 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- .

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fc9b9981636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7fc9b9981850]

Ignoring malparsed backtrace: 2021-03-11T17:33:18.583 INFO:teuthology.orchestra.run.gibba037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
, 2021-03-11T17:33:18.590 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight
, 2021-03-11T17:33:18.597 INFO:teuthology.orchestra.run.gibba017.stdout:osd.1: {}

Ignoring malparsed backtrace: 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.5191000000000002e-05
, 2021-03-11T17:33:18.926 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:18.927 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.488966+0000",
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 4.0315999999999998e-05
, 2021-03-11T17:33:18.949 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.508886+0000",
, 2021-03-11T17:33:18.971 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 2.1806000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",
, 2021-03-11T17:33:18.993 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.525921+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.015 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.037 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 8.6147e-05
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.038 INFO:teuthology.orchestra.run.gibba017.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:33:19.059 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.649523+0000",
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 0.000118886
, 2021-03-11T17:33:19.060 INFO:teuthology.orchestra.run.gibba017.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:33:19.081 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.685208+0000",
, 2021-03-11T17:33:19.082 INFO:teuthology.orchestra.run.gibba017.stdout:                        "duration": 7.2374999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",
, 2021-03-11T17:33:19.106 INFO:teuthology.orchestra.run.gibba017.stdout:                        "time": "2021-03-11T17:33:15.714711+0000",

Ignoring malparsed backtrace: 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    },
, 2021-03-11T17:33:19.132 INFO:teuthology.orchestra.run.gibba017.stdout:                    {
, 2021-03-11T17:33:19.133 INFO:teuthology.orchestra.run.gibba017.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:33:22.454+0000 7f8d2c5d0700 -1 WARNING: all dangerous and experimental features are enabled.
, set pool 1 pgp_num to 41
, 2021-03-11T17:33:23.357 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd

Ignoring malparsed backtrace: 2021-03-11T17:33:41.894 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":612,"fsid":"a21ff8f0-d934-473c-8fc0-01ea08a5489d","created":"2021-03-11T17:23:02.442549+0000","modified":"2021-03-11T17:33:41.423164+0000","last_up_change":"2021-03-11T17:33:14.440601+0000","last_in_change":"2021-03-11T17:33:22.334304+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:23:11.576521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":41,"pg_placement_num":41,"pg_placement_num_target":41,"pg_num_target":41,"pg_num_pending":41,"last_pg_merge_meta":{"source_pgid":"1.15","ready_epoch":238,"last_epoch_started":237,"last_epoch_clean":237,"source_version":"0'0","target_version":"0'0"},"last_change":"594","last_force_op_resend":"0","last_force_op_resend_prenautilus":"238","last_force_op_resend_preluminous":"544","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:23:12.713738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"2.8","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"595","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:23:24.612176+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":20,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":20,"last_pg_merge_meta":{"source_pgid":"3.14","ready_epoch":611,"last_epoch_started":610,"last_epoch_clean":610,"source_version":"590'1760","target_version":"566'1467"},"last_change":"611","last_force_op_resend":"0","last_force_op_resend_prenautilus":"611","last_force_op_resend_preluminous":"609","auid":0,"snap_mode":"selfmanaged","snap_seq":485,"snap_epoch":565,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"3bbe5845-5ddc-463f-bb23-ef8c12669fc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":530,"up_from":585,"up_thru":603,"down_at":531,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6802","nonce":101458},{"type":"v1","addr":"172.21.2.117:6803","nonce":101458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":101458},{"type":"v1","addr":"172.21.2.117:6807","nonce":101458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6813","nonce":101458},{"type":"v1","addr":"172.21.2.117:6815","nonce":101458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6809","nonce":101458},{"type":"v1","addr":"172.21.2.117:6811","nonce":101458}]},"public_addr":"172.21.2.117:6803/101458","cluster_addr":"172.21.2.117:6807/101458","heartbeat_back_addr":"172.21.2.117:6815/101458","heartbeat_front_addr":"172.21.2.117:6811/101458","state":["exists","up"]},{"osd":1,"uuid":"70d47745-6d00-4892-898c-a65f11520e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":611,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6819","nonce":35166},{"type":"v1","addr":"172.21.2.117:6821","nonce":35166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6823","nonce":35166},{"type":"v1","addr":"172.21.2.117:6825","nonce":35166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":35166},{"type":"v1","addr":"172.21.2.117:6831","nonce":35166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6827","nonce":35166},{"type":"v1","addr":"172.21.2.117:6829","nonce":35166}]},"public_addr":"172.21.2.117:6821/35166","cluster_addr":"172.21.2.117:6825/35166","heartbeat_back_addr":"172.21.2.117:6831/35166","heartbeat_front_addr":"172.21.2.117:6829/35166","state":["exists","up"]},{"osd":2,"uuid":"fb0542a4-1506-45ac-85f5-b2840d7ac488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":597,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":35167},{"type":"v1","addr":"172.21.2.117:6800","nonce":35167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":35167},{"type":"v1","addr":"172.21.2.117:6806","nonce":35167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":35167},{"type":"v1","addr":"172.21.2.117:6814","nonce":35167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6808","nonce":35167},{"type":"v1","addr":"172.21.2.117:6810","nonce":35167}]},"public_addr":"172.21.2.117:6800/35167","cluster_addr":"172.21.2.117:6806/35167","heartbeat_back_addr":"172.21.2.117:6814/35167","heartbeat_front_addr":"172.21.2.117:6810/35167","state":["exists","up"]},{"osd":3,"uuid":"fcf3328a-38fd-4049-b916-139a8c915fd8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":598,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":35168},{"type":"v1","addr":"172.21.2.117:6817","nonce":35168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":35168},{"type":"v1","addr":"172.21.2.117:6820","nonce":35168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":35168},{"type":"v1","addr":"172.21.2.117:6828","nonce":35168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":35168},{"type":"v1","addr":"172.21.2.117:6824","nonce":35168}]},"public_addr":"172.21.2.117:6817/35168","cluster_addr":"172.21.2.117:6820/35168","heartbeat_back_addr":"172.21.2.117:6828/35168","heartbeat_front_addr":"172.21.2.117:6824/35168","state":["exists","up"]},{"osd":4,"uuid":"8fb97983-57f2-4086-b856-4c8cab782051","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":599,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6800","nonce":35114},{"type":"v1","addr":"172.21.2.137:6801","nonce":35114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6802","nonce":35114},{"type":"v1","addr":"172.21.2.137:6803","nonce":35114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6806","nonce":35114},{"type":"v1","addr":"172.21.2.137:6807","nonce":35114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6804","nonce":35114},{"type":"v1","addr":"172.21.2.137:6805","nonce":35114}]},"public_addr":"172.21.2.137:6801/35114","cluster_addr":"172.21.2.137:6803/35114","heartbeat_back_addr":"172.21.2.137:6807/35114","heartbeat_front_addr":"172.21.2.137:6805/35114","state":["exists","up"]},{"osd":5,"uuid":"d635fd2e-4bb8-4965-82a9-e1c3b5db179f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":607,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6808","nonce":35115},{"type":"v1","addr":"172.21.2.137:6809","nonce":35115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6810","nonce":35115},{"type":"v1","addr":"172.21.2.137:6811","nonce":35115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6814","nonce":35115},{"type":"v1","addr":"172.21.2.137:6815","nonce":35115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6812","nonce":35115},{"type":"v1","addr":"172.21.2.137:6813","nonce":35115}]},"public_addr":"172.21.2.137:6809/35115","cluster_addr":"172.21.2.137:6811/35115","heartbeat_back_addr":"172.21.2.137:6815/35115","heartbeat_front_addr":"172.21.2.137:6813/35115","state":["exists","up"]},{"osd":6,"uuid":"427e3526-94f6-4820-8e03-86960993a78f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":297,"up_from":335,"up_thru":607,"down_at":298,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6816","nonce":57643},{"type":"v1","addr":"172.21.2.137:6817","nonce":57643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6818","nonce":57643},{"type":"v1","addr":"172.21.2.137:6819","nonce":57643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6823","nonce":57643},{"type":"v1","addr":"172.21.2.137:6825","nonce":57643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6820","nonce":57643},{"type":"v1","addr":"172.21.2.137:6821","nonce":57643}]},"public_addr":"172.21.2.137:6817/57643","cluster_addr":"172.21.2.137:6819/57643","heartbeat_back_addr":"172.21.2.137:6825/57643","heartbeat_front_addr":"172.21.2.137:6821/57643","state":["exists","up"]},{"osd":7,"uuid":"27250a3c-2474-4d7d-a2dd-fc2751e3f853","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":257,"up_from":274,"up_thru":607,"down_at":258,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6822","nonce":53803},{"type":"v1","addr":"172.21.2.137:6824","nonce":53803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6826","nonce":53803},{"type":"v1","addr":"172.21.2.137:6827","nonce":53803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6830","nonce":53803},{"type":"v1","addr":"172.21.2.137:6831","nonce":53803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.137:6828","nonce":53803},{"type":"v1","addr":"172.21.2.137:6829","nonce":53803}]},"public_addr":"172.21.2.137:6824/53803","cluster_addr":"172.21.2.137:6827/53803","heartbeat_back_addr":"172.21.2.137:6831/53803","heartbeat_front_addr":"172.21.2.137:6829/53803","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.187728+0000","dead_epoch":531},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.282348+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:15.265358+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:13.009966+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.849349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:14.470760+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.749489+0000","dead_epoch":298},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:33:12.855663+0000","dead_epoch":258}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"1.14","mappings":[{"from":2,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:33:41.905 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats
, 2021-03-11T17:33:41.994+0000 7ffa87322700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:33:48.000 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout
, 2021-03-11T17:33:48.001 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-03-11T17:33:48.087+0000 7eff86df1700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:05.277 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:34:05.364+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.
, 2021-03-11T17:34:05.375+0000 7f20f01b7700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2021-03-11T17:33:46.301750+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=46, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.094 INFO:tasks.ceph:pgid 1.12 last_scrub_stamp 2021-03-11T17:33:33.190052+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=33, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:34:08.095 INFO:tasks.ceph:pgid 1.13 last_scrub_stamp 2021-03-11T17:33:45.993208+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=33, tm_sec=45, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=34, tm_sec=3, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:35:09.132 INFO:teuthology.misc:Shutting down mds daemons...
, 2021-03-11T17:35:09.133 INFO:teuthology.misc:Shutting down osd daemons...
, 2021-03-11T17:35:09.133 DEBUG:tasks.ceph.osd.0:waiting for process to exit

Ignoring malparsed backtrace: 2021-03-11T17:36:56.988 DEBUG:teuthology.orchestra.run.gibba017:> sudo tar cz -f - -C /var/log/ceph -- .
, 2021-03-11T17:37:24.809 DEBUG:teuthology.misc:Transferring archived files from gibba037:/var/log/ceph to /home/teuthworker/archive/kchai-2021-03-11_16:25:48-rados-wip-kefu-testing-2021-03-11-2158-distro-basic-gibba/5956183/remote/gibba037/log
, 2021-03-11T17:37:24.814 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- .

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7368837636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f7368837850]

Ignoring malparsed backtrace: 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:45.231 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.678589+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.292 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 4.2830000000000003e-06
, 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.345 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00043970499999999998
, 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.393 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:        {
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDECSubOpWrite(3.1ds0 158/157 ECSubWrite(tid=583, reqid=osd.5.0:583, at_version=158'102, trim_to=0'0, roll_forward_to=158'100))",
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:42:05.626282+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.459 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.049485+0000",
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0079951829999999995
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.021484904999999999,
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:28.529859+0000",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00020556799999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.529752+0000",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7636999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.617726+0000",
, 2021-03-11T17:42:45.563 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.089910559000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:45.591 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.666220+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00076042099999999995
, 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.665 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "started",
, 2021-03-11T17:42:46.051 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:12.547443+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                "client_info": {
, 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client": "client.4403",
, 2021-03-11T17:42:46.135 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.187 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.054735736
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.804747+0000",
, 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.37268568200000002
, 2021-03-11T17:42:46.276 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "initiated",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.524942+0000",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.525024+0000",
, 2021-03-11T17:42:46.332 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0031246450000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:37.455666+0000",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.91714987000000003

Ignoring malparsed backtrace: 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:46.403 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.543441+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.461 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 5.9699999999999996e-06
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace: 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.932 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:40:22.522708+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.9399999999999996e-06
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0092418210000000008
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:33.540464+0000",
, 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.5289999999999997e-06
, 2021-03-11T17:42:47.136 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.064631+0000",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.279816+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.265 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "description": "MOSDECSubOpRead(3.bs1 186/157 ECSubRead(tid=607, to_read={3:d64931e1:::gibba02043757-32:head=0,1331200,0}, subchunks={3:d64931e1:::gibba02043757-32:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:33.546160+0000",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 13.335403349,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.22042878499999999
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.672 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 1.8488999999999999e-05
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:45.527339+0000",
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.787 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:54.668498+0000",
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0052109050000000001
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1650
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.625151+0000",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.3100000000000006e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:06.092621+0000",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.9757000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:21.632581+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:21.632690+0000",
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 25.993322320000001,
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 3.888350532,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.965 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00022072899999999999
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.328 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 4.2470000000000002e-06
, 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.351 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.372 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626293+0000",
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.7078e-05
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.400 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626407+0000",
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.5360000000000001e-06
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.652389+0000",
, 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.027341317
, 2021-03-11T17:42:48.425 INFO:teuthology.orchestra.run.gibba020.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.672256+0000",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.024543906000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.704777+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.492 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00044892499999999998
, 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.517 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:41.094589+0000",
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.000145088
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.013825456999999999,
, 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:48.561 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.893503+0000",
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.918 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.178077017,
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.717149+0000",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.176429103

Ignoring malparsed backtrace: 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:48.969 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.742446+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.993 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:            }
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.023 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.074885183999999994
, 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.050 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:27.770791+0000",
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.100 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:28.102429+0000",
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0010558290000000001
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:59.771774+0000",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.7230000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.156 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.509165+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.513 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.153071+0000",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00064247600000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.898208932,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.096293769000000001,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.702543+0000",
, 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.795110295,
, 2021-03-11T17:42:49.563 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.111779344,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.596 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDPGPushReply(3.4s0 86/85 [PushReplyOp(3:2cd859fe:::gibba02043757-45:2e)])",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.819625+0000",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.678027782,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.2030000000000003e-05
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.649 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:12.380655+0000",
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 6.3300000000000004e-06
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.678 INFO:teuthology.orchestra.run.gibba036.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1475
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.703 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "all_read",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620324+0000",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.1453e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620503+0000",
, 2021-03-11T17:42:49.730 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0015644330000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:10.628567+0000",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0061397810000000004

Ignoring malparsed backtrace: 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.489268+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.797 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:51.965903+0000",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0032776649999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:52.741147+0000",
, 2021-03-11T17:42:50.159 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.77191896699999996

Ignoring malparsed backtrace: 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:28.615358+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.031812119999999999
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1686
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                },
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.625063+0000",
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.5679000000000002e-05
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.627796+0000",
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.351e-05
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.357 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.054156+0000",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9321e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "initiated",
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526387+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting for peered",
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526493+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.427 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.676370+0000",
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.002219431
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:36.963203+0000",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7416000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.648660+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.655900+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.860 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.882 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.930 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.075e-06
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.7482e-05
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.653925+0000",
, 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9043999999999999e-05
, 2021-03-11T17:42:50.980 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.724300+0000",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.8502e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:40.699325+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.156 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:{
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:    "ops": [],

Ignoring malparsed backtrace: 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1080
, 2021-03-11T17:42:51.383 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.529651+0000",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 7.165e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:51.429 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.474 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.452e-06
, 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.497 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.518 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.001181545
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.542310+0000",
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4488e-05
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.563 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.59616179700000005,
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.585 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:46.514706+0000",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4453000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.607 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.627664+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.629 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:06.093788+0000",
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 45.263869956000001,
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.255713739,

Ignoring malparsed backtrace: 2021-03-11T17:43:08.320 INFO:teuthology.orchestra.run.gibba020.stdout:
, 2021-03-11T17:43:08.321 INFO:teuthology.orchestra.run.gibba020.stdout:{"epoch":220,"fsid":"060c3692-b38e-46ba-a56d-20dacac6db00","created":"2021-03-11T17:39:24.709482+0000","modified":"2021-03-11T17:43:07.852798+0000","last_up_change":"2021-03-11T17:41:10.517652+0000","last_in_change":"2021-03-11T17:42:44.555830+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:39:33.822798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"119","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:39:35.121814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":38,"pg_placement_num":38,"pg_placement_num_target":38,"pg_num_target":38,"pg_num_pending":38,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"137","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:39:46.860506+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"157","auid":0,"snap_mode":"selfmanaged","snap_seq":168,"snap_epoch":198,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"isaprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5fb4eb39-9240-4477-a421-f0fc71779ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6800","nonce":35152},{"type":"v1","addr":"172.21.2.120:6801","nonce":35152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":35152},{"type":"v1","addr":"172.21.2.120:6803","nonce":35152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6806","nonce":35152},{"type":"v1","addr":"172.21.2.120:6807","nonce":35152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":35152},{"type":"v1","addr":"172.21.2.120:6805","nonce":35152}]},"public_addr":"172.21.2.120:6801/35152","cluster_addr":"172.21.2.120:6803/35152","heartbeat_back_addr":"172.21.2.120:6807/35152","heartbeat_front_addr":"172.21.2.120:6805/35152","state":["exists","up"]},{"osd":1,"uuid":"2d2ff228-8155-47d4-91bb-79f0ed2f526f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":35153},{"type":"v1","addr":"172.21.2.120:6809","nonce":35153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":35153},{"type":"v1","addr":"172.21.2.120:6811","nonce":35153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":35153},{"type":"v1","addr":"172.21.2.120:6815","nonce":35153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6812","nonce":35153},{"type":"v1","addr":"172.21.2.120:6813","nonce":35153}]},"public_addr":"172.21.2.120:6809/35153","cluster_addr":"172.21.2.120:6811/35153","heartbeat_back_addr":"172.21.2.120:6815/35153","heartbeat_front_addr":"172.21.2.120:6813/35153","state":["exists","up"]},{"osd":2,"uuid":"4e4493e3-f491-477d-bf09-7dc3a460ec4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":35154},{"type":"v1","addr":"172.21.2.120:6817","nonce":35154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6818","nonce":35154},{"type":"v1","addr":"172.21.2.120:6819","nonce":35154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6826","nonce":35154},{"type":"v1","addr":"172.21.2.120:6828","nonce":35154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":35154},{"type":"v1","addr":"172.21.2.120:6824","nonce":35154}]},"public_addr":"172.21.2.120:6817/35154","cluster_addr":"172.21.2.120:6819/35154","heartbeat_back_addr":"172.21.2.120:6828/35154","heartbeat_front_addr":"172.21.2.120:6824/35154","state":["exists","up"]},{"osd":3,"uuid":"d641d703-a808-437c-ac2d-db9d508cb1e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":103,"up_thru":200,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":53119},{"type":"v1","addr":"172.21.2.120:6822","nonce":53119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6823","nonce":53119},{"type":"v1","addr":"172.21.2.120:6825","nonce":53119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":53119},{"type":"v1","addr":"172.21.2.120:6831","nonce":53119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6827","nonce":53119},{"type":"v1","addr":"172.21.2.120:6829","nonce":53119}]},"public_addr":"172.21.2.120:6822/53119","cluster_addr":"172.21.2.120:6825/53119","heartbeat_back_addr":"172.21.2.120:6831/53119","heartbeat_front_addr":"172.21.2.120:6829/53119","state":["exists","up"]},{"osd":4,"uuid":"f2ba6f4b-e144-4b2c-9100-5d8c6e6e2680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6800","nonce":35127},{"type":"v1","addr":"172.21.2.136:6801","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6802","nonce":35127},{"type":"v1","addr":"172.21.2.136:6803","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6806","nonce":35127},{"type":"v1","addr":"172.21.2.136:6807","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6804","nonce":35127},{"type":"v1","addr":"172.21.2.136:6805","nonce":35127}]},"public_addr":"172.21.2.136:6801/35127","cluster_addr":"172.21.2.136:6803/35127","heartbeat_back_addr":"172.21.2.136:6807/35127","heartbeat_front_addr":"172.21.2.136:6805/35127","state":["exists","up"]},{"osd":5,"uuid":"04e6b644-75ed-4a4b-bad0-c3c6ec60db04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6808","nonce":35128},{"type":"v1","addr":"172.21.2.136:6809","nonce":35128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6810","nonce":35128},{"type":"v1","addr":"172.21.2.136:6811","nonce":35128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6814","nonce":35128},{"type":"v1","addr":"172.21.2.136:6815","nonce":35128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6812","nonce":35128},{"type":"v1","addr":"172.21.2.136:6813","nonce":35128}]},"public_addr":"172.21.2.136:6809/35128","cluster_addr":"172.21.2.136:6811/35128","heartbeat_back_addr":"172.21.2.136:6815/35128","heartbeat_front_addr":"172.21.2.136:6813/35128","state":["exists","up"]},{"osd":6,"uuid":"1bef0f21-9f6a-4580-8618-34b3c99bdd16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6816","nonce":35129},{"type":"v1","addr":"172.21.2.136:6817","nonce":35129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6818","nonce":35129},{"type":"v1","addr":"172.21.2.136:6819","nonce":35129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6823","nonce":35129},{"type":"v1","addr":"172.21.2.136:6825","nonce":35129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6820","nonce":35129},{"type":"v1","addr":"172.21.2.136:6821","nonce":35129}]},"public_addr":"172.21.2.136:6817/35129","cluster_addr":"172.21.2.136:6819/35129","heartbeat_back_addr":"172.21.2.136:6825/35129","heartbeat_front_addr":"172.21.2.136:6821/35129","state":["exists","up"]},{"osd":7,"uuid":"938c8347-9c90-48f7-bef9-fdb6aad88bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6822","nonce":35130},{"type":"v1","addr":"172.21.2.136:6824","nonce":35130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6826","nonce":35130},{"type":"v1","addr":"172.21.2.136:6827","nonce":35130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6830","nonce":35130},{"type":"v1","addr":"172.21.2.136:6831","nonce":35130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6828","nonce":35130},{"type":"v1","addr":"172.21.2.136:6829","nonce":35130}]},"public_addr":"172.21.2.136:6824/35130","cluster_addr":"172.21.2.136:6827/35130","heartbeat_back_addr":"172.21.2.136:6831/35130","heartbeat_front_addr":"172.21.2.136:6829/35130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.721854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:32.685787+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.339122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:09.743044+0000","dead_epoch":18},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.661501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.717093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:34.439797+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.618674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"2.21","osds":[6,7]}],"pg_upmap_items":[{"pgid":"1.8","mappings":[{"from":5,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1d","mappings":[{"from":4,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"isaprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"2","m":"1","name":"isaprofile","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:43:08.331 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats

Ignoring malparsed backtrace: 2021-03-11T17:43:16.547 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4
, instructed osd(s) 4 to deep-scrub
, 2021-03-11T17:43:16.771 INFO:tasks.ceph:Scrubbing osd.5

Ignoring malparsed backtrace: 2021-03-11T17:43:18.192 INFO:tasks.ceph:pgid 1.17 last_scrub_stamp 2021-03-11T17:42:44.288800+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.193 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2021-03-11T17:42:44.490732+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.194 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2021-03-11T17:43:06.994647+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=6, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:43:58.828 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2021-03-11T17:41:29.916342+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=41, tm_sec=29, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.829 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2021-03-11T17:42:22.136291+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.830 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-11T17:42:22.618589+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:44:20.744 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6
, 2021-03-11T17:44:20.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba036.front.sepia.ceph.com
, 2021-03-11T17:44:20.842 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7

Ignoring malparsed backtrace: 2021-03-11T17:44:40.229 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'
, 2021-03-11T17:44:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2021-03-11T17:44:40.252 DEBUG:teuthology.orchestra.run.gibba036:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7368837636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f7368837850]

Ignoring malparsed backtrace: 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:45.231 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.678589+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.292 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 4.2830000000000003e-06
, 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.345 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00043970499999999998
, 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.393 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:        {
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDECSubOpWrite(3.1ds0 158/157 ECSubWrite(tid=583, reqid=osd.5.0:583, at_version=158'102, trim_to=0'0, roll_forward_to=158'100))",
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:42:05.626282+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.459 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.049485+0000",
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0079951829999999995
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.021484904999999999,
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:28.529859+0000",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00020556799999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.529752+0000",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7636999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.617726+0000",
, 2021-03-11T17:42:45.563 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.089910559000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:45.591 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.666220+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00076042099999999995
, 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.665 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "started",
, 2021-03-11T17:42:46.051 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:12.547443+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                "client_info": {
, 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client": "client.4403",
, 2021-03-11T17:42:46.135 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.187 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.054735736
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.804747+0000",
, 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.37268568200000002
, 2021-03-11T17:42:46.276 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "initiated",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.524942+0000",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.525024+0000",
, 2021-03-11T17:42:46.332 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0031246450000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:37.455666+0000",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.91714987000000003

Ignoring malparsed backtrace: 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:46.403 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.543441+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.461 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 5.9699999999999996e-06
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace: 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.932 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:40:22.522708+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.9399999999999996e-06
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0092418210000000008
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:33.540464+0000",
, 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.5289999999999997e-06
, 2021-03-11T17:42:47.136 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.064631+0000",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.279816+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.265 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "description": "MOSDECSubOpRead(3.bs1 186/157 ECSubRead(tid=607, to_read={3:d64931e1:::gibba02043757-32:head=0,1331200,0}, subchunks={3:d64931e1:::gibba02043757-32:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:33.546160+0000",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 13.335403349,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.22042878499999999
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.672 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 1.8488999999999999e-05
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:45.527339+0000",
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.787 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:54.668498+0000",
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0052109050000000001
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1650
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.625151+0000",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.3100000000000006e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:06.092621+0000",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.9757000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:21.632581+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:21.632690+0000",
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 25.993322320000001,
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 3.888350532,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.965 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00022072899999999999
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.328 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 4.2470000000000002e-06
, 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.351 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.372 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626293+0000",
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.7078e-05
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.400 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626407+0000",
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.5360000000000001e-06
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.652389+0000",
, 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.027341317
, 2021-03-11T17:42:48.425 INFO:teuthology.orchestra.run.gibba020.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.672256+0000",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.024543906000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.704777+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.492 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00044892499999999998
, 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.517 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:41.094589+0000",
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.000145088
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.013825456999999999,
, 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:48.561 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.893503+0000",
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.918 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.178077017,
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.717149+0000",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.176429103

Ignoring malparsed backtrace: 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:48.969 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.742446+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.993 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:            }
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.023 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.074885183999999994
, 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.050 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:27.770791+0000",
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.100 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:28.102429+0000",
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0010558290000000001
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:59.771774+0000",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.7230000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.156 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.509165+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.513 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.153071+0000",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00064247600000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.898208932,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.096293769000000001,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.702543+0000",
, 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.795110295,
, 2021-03-11T17:42:49.563 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.111779344,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.596 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDPGPushReply(3.4s0 86/85 [PushReplyOp(3:2cd859fe:::gibba02043757-45:2e)])",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.819625+0000",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.678027782,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.2030000000000003e-05
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.649 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:12.380655+0000",
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 6.3300000000000004e-06
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.678 INFO:teuthology.orchestra.run.gibba036.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1475
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.703 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "all_read",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620324+0000",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.1453e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620503+0000",
, 2021-03-11T17:42:49.730 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0015644330000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:10.628567+0000",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0061397810000000004

Ignoring malparsed backtrace: 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.489268+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.797 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:51.965903+0000",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0032776649999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:52.741147+0000",
, 2021-03-11T17:42:50.159 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.77191896699999996

Ignoring malparsed backtrace: 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:28.615358+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.031812119999999999
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1686
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                },
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.625063+0000",
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.5679000000000002e-05
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.627796+0000",
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.351e-05
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.357 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.054156+0000",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9321e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "initiated",
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526387+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting for peered",
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526493+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.427 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.676370+0000",
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.002219431
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:36.963203+0000",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7416000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.648660+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.655900+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.860 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.882 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.930 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.075e-06
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.7482e-05
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.653925+0000",
, 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9043999999999999e-05
, 2021-03-11T17:42:50.980 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.724300+0000",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.8502e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:40.699325+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.156 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:{
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:    "ops": [],

Ignoring malparsed backtrace: 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1080
, 2021-03-11T17:42:51.383 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.529651+0000",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 7.165e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:51.429 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.474 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.452e-06
, 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.497 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.518 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.001181545
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.542310+0000",
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4488e-05
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.563 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.59616179700000005,
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.585 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:46.514706+0000",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4453000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.607 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.627664+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.629 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:06.093788+0000",
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 45.263869956000001,
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.255713739,

Ignoring malparsed backtrace: 2021-03-11T17:43:08.320 INFO:teuthology.orchestra.run.gibba020.stdout:
, 2021-03-11T17:43:08.321 INFO:teuthology.orchestra.run.gibba020.stdout:{"epoch":220,"fsid":"060c3692-b38e-46ba-a56d-20dacac6db00","created":"2021-03-11T17:39:24.709482+0000","modified":"2021-03-11T17:43:07.852798+0000","last_up_change":"2021-03-11T17:41:10.517652+0000","last_in_change":"2021-03-11T17:42:44.555830+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:39:33.822798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"119","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:39:35.121814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":38,"pg_placement_num":38,"pg_placement_num_target":38,"pg_num_target":38,"pg_num_pending":38,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"137","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:39:46.860506+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"157","auid":0,"snap_mode":"selfmanaged","snap_seq":168,"snap_epoch":198,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"isaprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5fb4eb39-9240-4477-a421-f0fc71779ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6800","nonce":35152},{"type":"v1","addr":"172.21.2.120:6801","nonce":35152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":35152},{"type":"v1","addr":"172.21.2.120:6803","nonce":35152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6806","nonce":35152},{"type":"v1","addr":"172.21.2.120:6807","nonce":35152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":35152},{"type":"v1","addr":"172.21.2.120:6805","nonce":35152}]},"public_addr":"172.21.2.120:6801/35152","cluster_addr":"172.21.2.120:6803/35152","heartbeat_back_addr":"172.21.2.120:6807/35152","heartbeat_front_addr":"172.21.2.120:6805/35152","state":["exists","up"]},{"osd":1,"uuid":"2d2ff228-8155-47d4-91bb-79f0ed2f526f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":35153},{"type":"v1","addr":"172.21.2.120:6809","nonce":35153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":35153},{"type":"v1","addr":"172.21.2.120:6811","nonce":35153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":35153},{"type":"v1","addr":"172.21.2.120:6815","nonce":35153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6812","nonce":35153},{"type":"v1","addr":"172.21.2.120:6813","nonce":35153}]},"public_addr":"172.21.2.120:6809/35153","cluster_addr":"172.21.2.120:6811/35153","heartbeat_back_addr":"172.21.2.120:6815/35153","heartbeat_front_addr":"172.21.2.120:6813/35153","state":["exists","up"]},{"osd":2,"uuid":"4e4493e3-f491-477d-bf09-7dc3a460ec4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":35154},{"type":"v1","addr":"172.21.2.120:6817","nonce":35154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6818","nonce":35154},{"type":"v1","addr":"172.21.2.120:6819","nonce":35154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6826","nonce":35154},{"type":"v1","addr":"172.21.2.120:6828","nonce":35154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":35154},{"type":"v1","addr":"172.21.2.120:6824","nonce":35154}]},"public_addr":"172.21.2.120:6817/35154","cluster_addr":"172.21.2.120:6819/35154","heartbeat_back_addr":"172.21.2.120:6828/35154","heartbeat_front_addr":"172.21.2.120:6824/35154","state":["exists","up"]},{"osd":3,"uuid":"d641d703-a808-437c-ac2d-db9d508cb1e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":103,"up_thru":200,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":53119},{"type":"v1","addr":"172.21.2.120:6822","nonce":53119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6823","nonce":53119},{"type":"v1","addr":"172.21.2.120:6825","nonce":53119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":53119},{"type":"v1","addr":"172.21.2.120:6831","nonce":53119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6827","nonce":53119},{"type":"v1","addr":"172.21.2.120:6829","nonce":53119}]},"public_addr":"172.21.2.120:6822/53119","cluster_addr":"172.21.2.120:6825/53119","heartbeat_back_addr":"172.21.2.120:6831/53119","heartbeat_front_addr":"172.21.2.120:6829/53119","state":["exists","up"]},{"osd":4,"uuid":"f2ba6f4b-e144-4b2c-9100-5d8c6e6e2680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6800","nonce":35127},{"type":"v1","addr":"172.21.2.136:6801","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6802","nonce":35127},{"type":"v1","addr":"172.21.2.136:6803","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6806","nonce":35127},{"type":"v1","addr":"172.21.2.136:6807","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6804","nonce":35127},{"type":"v1","addr":"172.21.2.136:6805","nonce":35127}]},"public_addr":"172.21.2.136:6801/35127","cluster_addr":"172.21.2.136:6803/35127","heartbeat_back_addr":"172.21.2.136:6807/35127","heartbeat_front_addr":"172.21.2.136:6805/35127","state":["exists","up"]},{"osd":5,"uuid":"04e6b644-75ed-4a4b-bad0-c3c6ec60db04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6808","nonce":35128},{"type":"v1","addr":"172.21.2.136:6809","nonce":35128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6810","nonce":35128},{"type":"v1","addr":"172.21.2.136:6811","nonce":35128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6814","nonce":35128},{"type":"v1","addr":"172.21.2.136:6815","nonce":35128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6812","nonce":35128},{"type":"v1","addr":"172.21.2.136:6813","nonce":35128}]},"public_addr":"172.21.2.136:6809/35128","cluster_addr":"172.21.2.136:6811/35128","heartbeat_back_addr":"172.21.2.136:6815/35128","heartbeat_front_addr":"172.21.2.136:6813/35128","state":["exists","up"]},{"osd":6,"uuid":"1bef0f21-9f6a-4580-8618-34b3c99bdd16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6816","nonce":35129},{"type":"v1","addr":"172.21.2.136:6817","nonce":35129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6818","nonce":35129},{"type":"v1","addr":"172.21.2.136:6819","nonce":35129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6823","nonce":35129},{"type":"v1","addr":"172.21.2.136:6825","nonce":35129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6820","nonce":35129},{"type":"v1","addr":"172.21.2.136:6821","nonce":35129}]},"public_addr":"172.21.2.136:6817/35129","cluster_addr":"172.21.2.136:6819/35129","heartbeat_back_addr":"172.21.2.136:6825/35129","heartbeat_front_addr":"172.21.2.136:6821/35129","state":["exists","up"]},{"osd":7,"uuid":"938c8347-9c90-48f7-bef9-fdb6aad88bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6822","nonce":35130},{"type":"v1","addr":"172.21.2.136:6824","nonce":35130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6826","nonce":35130},{"type":"v1","addr":"172.21.2.136:6827","nonce":35130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6830","nonce":35130},{"type":"v1","addr":"172.21.2.136:6831","nonce":35130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6828","nonce":35130},{"type":"v1","addr":"172.21.2.136:6829","nonce":35130}]},"public_addr":"172.21.2.136:6824/35130","cluster_addr":"172.21.2.136:6827/35130","heartbeat_back_addr":"172.21.2.136:6831/35130","heartbeat_front_addr":"172.21.2.136:6829/35130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.721854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:32.685787+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.339122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:09.743044+0000","dead_epoch":18},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.661501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.717093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:34.439797+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.618674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"2.21","osds":[6,7]}],"pg_upmap_items":[{"pgid":"1.8","mappings":[{"from":5,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1d","mappings":[{"from":4,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"isaprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"2","m":"1","name":"isaprofile","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:43:08.331 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats

Ignoring malparsed backtrace: 2021-03-11T17:43:16.547 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4
, instructed osd(s) 4 to deep-scrub
, 2021-03-11T17:43:16.771 INFO:tasks.ceph:Scrubbing osd.5

Ignoring malparsed backtrace: 2021-03-11T17:43:18.192 INFO:tasks.ceph:pgid 1.17 last_scrub_stamp 2021-03-11T17:42:44.288800+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.193 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2021-03-11T17:42:44.490732+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.194 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2021-03-11T17:43:06.994647+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=6, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:43:58.828 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2021-03-11T17:41:29.916342+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=41, tm_sec=29, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.829 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2021-03-11T17:42:22.136291+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.830 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-11T17:42:22.618589+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:44:20.744 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6
, 2021-03-11T17:44:20.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba036.front.sepia.ceph.com
, 2021-03-11T17:44:20.842 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7

Ignoring malparsed backtrace: 2021-03-11T17:44:40.229 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'
, 2021-03-11T17:44:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2021-03-11T17:44:40.252 DEBUG:teuthology.orchestra.run.gibba036:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7368837636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f7368837850]

Ignoring malparsed backtrace: 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:45.231 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.678589+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.292 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 4.2830000000000003e-06
, 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.345 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00043970499999999998
, 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.393 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:        {
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDECSubOpWrite(3.1ds0 158/157 ECSubWrite(tid=583, reqid=osd.5.0:583, at_version=158'102, trim_to=0'0, roll_forward_to=158'100))",
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:42:05.626282+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.459 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.049485+0000",
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0079951829999999995
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.021484904999999999,
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:28.529859+0000",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00020556799999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.529752+0000",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7636999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.617726+0000",
, 2021-03-11T17:42:45.563 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.089910559000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:45.591 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.666220+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00076042099999999995
, 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.665 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "started",
, 2021-03-11T17:42:46.051 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:12.547443+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                "client_info": {
, 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client": "client.4403",
, 2021-03-11T17:42:46.135 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.187 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.054735736
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.804747+0000",
, 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.37268568200000002
, 2021-03-11T17:42:46.276 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "initiated",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.524942+0000",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.525024+0000",
, 2021-03-11T17:42:46.332 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0031246450000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:37.455666+0000",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.91714987000000003

Ignoring malparsed backtrace: 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:46.403 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.543441+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.461 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 5.9699999999999996e-06
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace: 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.932 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:40:22.522708+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.9399999999999996e-06
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0092418210000000008
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:33.540464+0000",
, 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.5289999999999997e-06
, 2021-03-11T17:42:47.136 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.064631+0000",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.279816+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.265 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "description": "MOSDECSubOpRead(3.bs1 186/157 ECSubRead(tid=607, to_read={3:d64931e1:::gibba02043757-32:head=0,1331200,0}, subchunks={3:d64931e1:::gibba02043757-32:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:33.546160+0000",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 13.335403349,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.22042878499999999
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.672 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 1.8488999999999999e-05
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:45.527339+0000",
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.787 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:54.668498+0000",
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0052109050000000001
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1650
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.625151+0000",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.3100000000000006e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:06.092621+0000",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.9757000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:21.632581+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:21.632690+0000",
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 25.993322320000001,
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 3.888350532,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.965 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00022072899999999999
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.328 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 4.2470000000000002e-06
, 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.351 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.372 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626293+0000",
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.7078e-05
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.400 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626407+0000",
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.5360000000000001e-06
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.652389+0000",
, 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.027341317
, 2021-03-11T17:42:48.425 INFO:teuthology.orchestra.run.gibba020.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.672256+0000",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.024543906000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.704777+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.492 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00044892499999999998
, 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.517 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:41.094589+0000",
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.000145088
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.013825456999999999,
, 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:48.561 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.893503+0000",
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.918 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.178077017,
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.717149+0000",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.176429103

Ignoring malparsed backtrace: 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:48.969 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.742446+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.993 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:            }
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.023 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.074885183999999994
, 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.050 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:27.770791+0000",
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.100 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:28.102429+0000",
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0010558290000000001
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:59.771774+0000",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.7230000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.156 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.509165+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.513 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.153071+0000",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00064247600000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.898208932,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.096293769000000001,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.702543+0000",
, 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.795110295,
, 2021-03-11T17:42:49.563 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.111779344,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.596 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDPGPushReply(3.4s0 86/85 [PushReplyOp(3:2cd859fe:::gibba02043757-45:2e)])",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.819625+0000",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.678027782,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.2030000000000003e-05
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.649 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:12.380655+0000",
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 6.3300000000000004e-06
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.678 INFO:teuthology.orchestra.run.gibba036.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1475
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.703 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "all_read",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620324+0000",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.1453e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620503+0000",
, 2021-03-11T17:42:49.730 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0015644330000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:10.628567+0000",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0061397810000000004

Ignoring malparsed backtrace: 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.489268+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.797 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:51.965903+0000",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0032776649999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:52.741147+0000",
, 2021-03-11T17:42:50.159 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.77191896699999996

Ignoring malparsed backtrace: 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:28.615358+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.031812119999999999
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1686
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                },
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.625063+0000",
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.5679000000000002e-05
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.627796+0000",
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.351e-05
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.357 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.054156+0000",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9321e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "initiated",
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526387+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting for peered",
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526493+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.427 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.676370+0000",
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.002219431
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:36.963203+0000",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7416000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.648660+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.655900+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.860 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.882 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.930 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.075e-06
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.7482e-05
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.653925+0000",
, 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9043999999999999e-05
, 2021-03-11T17:42:50.980 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.724300+0000",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.8502e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:40.699325+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.156 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:{
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:    "ops": [],

Ignoring malparsed backtrace: 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1080
, 2021-03-11T17:42:51.383 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.529651+0000",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 7.165e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:51.429 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.474 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.452e-06
, 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.497 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.518 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.001181545
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.542310+0000",
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4488e-05
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.563 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.59616179700000005,
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.585 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:46.514706+0000",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4453000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.607 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.627664+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.629 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:06.093788+0000",
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 45.263869956000001,
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.255713739,

Ignoring malparsed backtrace: 2021-03-11T17:43:08.320 INFO:teuthology.orchestra.run.gibba020.stdout:
, 2021-03-11T17:43:08.321 INFO:teuthology.orchestra.run.gibba020.stdout:{"epoch":220,"fsid":"060c3692-b38e-46ba-a56d-20dacac6db00","created":"2021-03-11T17:39:24.709482+0000","modified":"2021-03-11T17:43:07.852798+0000","last_up_change":"2021-03-11T17:41:10.517652+0000","last_in_change":"2021-03-11T17:42:44.555830+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:39:33.822798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"119","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:39:35.121814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":38,"pg_placement_num":38,"pg_placement_num_target":38,"pg_num_target":38,"pg_num_pending":38,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"137","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:39:46.860506+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"157","auid":0,"snap_mode":"selfmanaged","snap_seq":168,"snap_epoch":198,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"isaprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5fb4eb39-9240-4477-a421-f0fc71779ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6800","nonce":35152},{"type":"v1","addr":"172.21.2.120:6801","nonce":35152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":35152},{"type":"v1","addr":"172.21.2.120:6803","nonce":35152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6806","nonce":35152},{"type":"v1","addr":"172.21.2.120:6807","nonce":35152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":35152},{"type":"v1","addr":"172.21.2.120:6805","nonce":35152}]},"public_addr":"172.21.2.120:6801/35152","cluster_addr":"172.21.2.120:6803/35152","heartbeat_back_addr":"172.21.2.120:6807/35152","heartbeat_front_addr":"172.21.2.120:6805/35152","state":["exists","up"]},{"osd":1,"uuid":"2d2ff228-8155-47d4-91bb-79f0ed2f526f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":35153},{"type":"v1","addr":"172.21.2.120:6809","nonce":35153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":35153},{"type":"v1","addr":"172.21.2.120:6811","nonce":35153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":35153},{"type":"v1","addr":"172.21.2.120:6815","nonce":35153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6812","nonce":35153},{"type":"v1","addr":"172.21.2.120:6813","nonce":35153}]},"public_addr":"172.21.2.120:6809/35153","cluster_addr":"172.21.2.120:6811/35153","heartbeat_back_addr":"172.21.2.120:6815/35153","heartbeat_front_addr":"172.21.2.120:6813/35153","state":["exists","up"]},{"osd":2,"uuid":"4e4493e3-f491-477d-bf09-7dc3a460ec4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":35154},{"type":"v1","addr":"172.21.2.120:6817","nonce":35154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6818","nonce":35154},{"type":"v1","addr":"172.21.2.120:6819","nonce":35154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6826","nonce":35154},{"type":"v1","addr":"172.21.2.120:6828","nonce":35154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":35154},{"type":"v1","addr":"172.21.2.120:6824","nonce":35154}]},"public_addr":"172.21.2.120:6817/35154","cluster_addr":"172.21.2.120:6819/35154","heartbeat_back_addr":"172.21.2.120:6828/35154","heartbeat_front_addr":"172.21.2.120:6824/35154","state":["exists","up"]},{"osd":3,"uuid":"d641d703-a808-437c-ac2d-db9d508cb1e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":103,"up_thru":200,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":53119},{"type":"v1","addr":"172.21.2.120:6822","nonce":53119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6823","nonce":53119},{"type":"v1","addr":"172.21.2.120:6825","nonce":53119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":53119},{"type":"v1","addr":"172.21.2.120:6831","nonce":53119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6827","nonce":53119},{"type":"v1","addr":"172.21.2.120:6829","nonce":53119}]},"public_addr":"172.21.2.120:6822/53119","cluster_addr":"172.21.2.120:6825/53119","heartbeat_back_addr":"172.21.2.120:6831/53119","heartbeat_front_addr":"172.21.2.120:6829/53119","state":["exists","up"]},{"osd":4,"uuid":"f2ba6f4b-e144-4b2c-9100-5d8c6e6e2680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6800","nonce":35127},{"type":"v1","addr":"172.21.2.136:6801","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6802","nonce":35127},{"type":"v1","addr":"172.21.2.136:6803","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6806","nonce":35127},{"type":"v1","addr":"172.21.2.136:6807","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6804","nonce":35127},{"type":"v1","addr":"172.21.2.136:6805","nonce":35127}]},"public_addr":"172.21.2.136:6801/35127","cluster_addr":"172.21.2.136:6803/35127","heartbeat_back_addr":"172.21.2.136:6807/35127","heartbeat_front_addr":"172.21.2.136:6805/35127","state":["exists","up"]},{"osd":5,"uuid":"04e6b644-75ed-4a4b-bad0-c3c6ec60db04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6808","nonce":35128},{"type":"v1","addr":"172.21.2.136:6809","nonce":35128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6810","nonce":35128},{"type":"v1","addr":"172.21.2.136:6811","nonce":35128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6814","nonce":35128},{"type":"v1","addr":"172.21.2.136:6815","nonce":35128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6812","nonce":35128},{"type":"v1","addr":"172.21.2.136:6813","nonce":35128}]},"public_addr":"172.21.2.136:6809/35128","cluster_addr":"172.21.2.136:6811/35128","heartbeat_back_addr":"172.21.2.136:6815/35128","heartbeat_front_addr":"172.21.2.136:6813/35128","state":["exists","up"]},{"osd":6,"uuid":"1bef0f21-9f6a-4580-8618-34b3c99bdd16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6816","nonce":35129},{"type":"v1","addr":"172.21.2.136:6817","nonce":35129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6818","nonce":35129},{"type":"v1","addr":"172.21.2.136:6819","nonce":35129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6823","nonce":35129},{"type":"v1","addr":"172.21.2.136:6825","nonce":35129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6820","nonce":35129},{"type":"v1","addr":"172.21.2.136:6821","nonce":35129}]},"public_addr":"172.21.2.136:6817/35129","cluster_addr":"172.21.2.136:6819/35129","heartbeat_back_addr":"172.21.2.136:6825/35129","heartbeat_front_addr":"172.21.2.136:6821/35129","state":["exists","up"]},{"osd":7,"uuid":"938c8347-9c90-48f7-bef9-fdb6aad88bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6822","nonce":35130},{"type":"v1","addr":"172.21.2.136:6824","nonce":35130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6826","nonce":35130},{"type":"v1","addr":"172.21.2.136:6827","nonce":35130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6830","nonce":35130},{"type":"v1","addr":"172.21.2.136:6831","nonce":35130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6828","nonce":35130},{"type":"v1","addr":"172.21.2.136:6829","nonce":35130}]},"public_addr":"172.21.2.136:6824/35130","cluster_addr":"172.21.2.136:6827/35130","heartbeat_back_addr":"172.21.2.136:6831/35130","heartbeat_front_addr":"172.21.2.136:6829/35130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.721854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:32.685787+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.339122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:09.743044+0000","dead_epoch":18},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.661501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.717093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:34.439797+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.618674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"2.21","osds":[6,7]}],"pg_upmap_items":[{"pgid":"1.8","mappings":[{"from":5,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1d","mappings":[{"from":4,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"isaprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"2","m":"1","name":"isaprofile","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:43:08.331 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats

Ignoring malparsed backtrace: 2021-03-11T17:43:16.547 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4
, instructed osd(s) 4 to deep-scrub
, 2021-03-11T17:43:16.771 INFO:tasks.ceph:Scrubbing osd.5

Ignoring malparsed backtrace: 2021-03-11T17:43:18.192 INFO:tasks.ceph:pgid 1.17 last_scrub_stamp 2021-03-11T17:42:44.288800+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.193 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2021-03-11T17:42:44.490732+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.194 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2021-03-11T17:43:06.994647+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=6, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:43:58.828 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2021-03-11T17:41:29.916342+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=41, tm_sec=29, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.829 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2021-03-11T17:42:22.136291+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.830 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-11T17:42:22.618589+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:44:20.744 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6
, 2021-03-11T17:44:20.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba036.front.sepia.ceph.com
, 2021-03-11T17:44:20.842 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7

Ignoring malparsed backtrace: 2021-03-11T17:44:40.229 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'
, 2021-03-11T17:44:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2021-03-11T17:44:40.252 DEBUG:teuthology.orchestra.run.gibba036:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7368837636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f7368837850]

Ignoring malparsed backtrace: 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:45.231 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.678589+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.292 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 4.2830000000000003e-06
, 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.345 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00043970499999999998
, 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.393 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:        {
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDECSubOpWrite(3.1ds0 158/157 ECSubWrite(tid=583, reqid=osd.5.0:583, at_version=158'102, trim_to=0'0, roll_forward_to=158'100))",
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:42:05.626282+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.459 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.049485+0000",
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0079951829999999995
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.021484904999999999,
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:28.529859+0000",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00020556799999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.529752+0000",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7636999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.617726+0000",
, 2021-03-11T17:42:45.563 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.089910559000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:45.591 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.666220+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00076042099999999995
, 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.665 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "started",
, 2021-03-11T17:42:46.051 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:12.547443+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                "client_info": {
, 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client": "client.4403",
, 2021-03-11T17:42:46.135 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.187 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.054735736
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.804747+0000",
, 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.37268568200000002
, 2021-03-11T17:42:46.276 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "initiated",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.524942+0000",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.525024+0000",
, 2021-03-11T17:42:46.332 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0031246450000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:37.455666+0000",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.91714987000000003

Ignoring malparsed backtrace: 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:46.403 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.543441+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.461 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 5.9699999999999996e-06
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace: 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.932 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:40:22.522708+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.9399999999999996e-06
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0092418210000000008
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:33.540464+0000",
, 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.5289999999999997e-06
, 2021-03-11T17:42:47.136 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.064631+0000",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.279816+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.265 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "description": "MOSDECSubOpRead(3.bs1 186/157 ECSubRead(tid=607, to_read={3:d64931e1:::gibba02043757-32:head=0,1331200,0}, subchunks={3:d64931e1:::gibba02043757-32:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:33.546160+0000",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 13.335403349,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.22042878499999999
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.672 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 1.8488999999999999e-05
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:45.527339+0000",
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.787 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:54.668498+0000",
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0052109050000000001
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1650
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.625151+0000",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.3100000000000006e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:06.092621+0000",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.9757000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:21.632581+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:21.632690+0000",
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 25.993322320000001,
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 3.888350532,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.965 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00022072899999999999
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.328 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 4.2470000000000002e-06
, 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.351 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.372 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626293+0000",
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.7078e-05
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.400 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626407+0000",
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.5360000000000001e-06
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.652389+0000",
, 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.027341317
, 2021-03-11T17:42:48.425 INFO:teuthology.orchestra.run.gibba020.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.672256+0000",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.024543906000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.704777+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.492 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00044892499999999998
, 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.517 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:41.094589+0000",
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.000145088
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.013825456999999999,
, 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:48.561 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.893503+0000",
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.918 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.178077017,
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.717149+0000",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.176429103

Ignoring malparsed backtrace: 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:48.969 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.742446+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.993 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:            }
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.023 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.074885183999999994
, 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.050 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:27.770791+0000",
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.100 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:28.102429+0000",
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0010558290000000001
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:59.771774+0000",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.7230000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.156 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.509165+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.513 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.153071+0000",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00064247600000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.898208932,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.096293769000000001,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.702543+0000",
, 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.795110295,
, 2021-03-11T17:42:49.563 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.111779344,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.596 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDPGPushReply(3.4s0 86/85 [PushReplyOp(3:2cd859fe:::gibba02043757-45:2e)])",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.819625+0000",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.678027782,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.2030000000000003e-05
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.649 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:12.380655+0000",
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 6.3300000000000004e-06
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.678 INFO:teuthology.orchestra.run.gibba036.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1475
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.703 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "all_read",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620324+0000",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.1453e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620503+0000",
, 2021-03-11T17:42:49.730 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0015644330000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:10.628567+0000",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0061397810000000004

Ignoring malparsed backtrace: 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.489268+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.797 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:51.965903+0000",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0032776649999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:52.741147+0000",
, 2021-03-11T17:42:50.159 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.77191896699999996

Ignoring malparsed backtrace: 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:28.615358+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.031812119999999999
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1686
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                },
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.625063+0000",
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.5679000000000002e-05
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.627796+0000",
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.351e-05
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.357 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.054156+0000",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9321e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "initiated",
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526387+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting for peered",
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526493+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.427 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.676370+0000",
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.002219431
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:36.963203+0000",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7416000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.648660+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.655900+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.860 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.882 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.930 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.075e-06
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.7482e-05
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.653925+0000",
, 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9043999999999999e-05
, 2021-03-11T17:42:50.980 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.724300+0000",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.8502e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:40.699325+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.156 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:{
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:    "ops": [],

Ignoring malparsed backtrace: 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1080
, 2021-03-11T17:42:51.383 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.529651+0000",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 7.165e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:51.429 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.474 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.452e-06
, 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.497 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.518 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.001181545
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.542310+0000",
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4488e-05
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.563 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.59616179700000005,
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.585 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:46.514706+0000",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4453000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.607 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.627664+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.629 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:06.093788+0000",
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 45.263869956000001,
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.255713739,

Ignoring malparsed backtrace: 2021-03-11T17:43:08.320 INFO:teuthology.orchestra.run.gibba020.stdout:
, 2021-03-11T17:43:08.321 INFO:teuthology.orchestra.run.gibba020.stdout:{"epoch":220,"fsid":"060c3692-b38e-46ba-a56d-20dacac6db00","created":"2021-03-11T17:39:24.709482+0000","modified":"2021-03-11T17:43:07.852798+0000","last_up_change":"2021-03-11T17:41:10.517652+0000","last_in_change":"2021-03-11T17:42:44.555830+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:39:33.822798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"119","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:39:35.121814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":38,"pg_placement_num":38,"pg_placement_num_target":38,"pg_num_target":38,"pg_num_pending":38,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"137","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:39:46.860506+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"157","auid":0,"snap_mode":"selfmanaged","snap_seq":168,"snap_epoch":198,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"isaprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5fb4eb39-9240-4477-a421-f0fc71779ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6800","nonce":35152},{"type":"v1","addr":"172.21.2.120:6801","nonce":35152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":35152},{"type":"v1","addr":"172.21.2.120:6803","nonce":35152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6806","nonce":35152},{"type":"v1","addr":"172.21.2.120:6807","nonce":35152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":35152},{"type":"v1","addr":"172.21.2.120:6805","nonce":35152}]},"public_addr":"172.21.2.120:6801/35152","cluster_addr":"172.21.2.120:6803/35152","heartbeat_back_addr":"172.21.2.120:6807/35152","heartbeat_front_addr":"172.21.2.120:6805/35152","state":["exists","up"]},{"osd":1,"uuid":"2d2ff228-8155-47d4-91bb-79f0ed2f526f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":35153},{"type":"v1","addr":"172.21.2.120:6809","nonce":35153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":35153},{"type":"v1","addr":"172.21.2.120:6811","nonce":35153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":35153},{"type":"v1","addr":"172.21.2.120:6815","nonce":35153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6812","nonce":35153},{"type":"v1","addr":"172.21.2.120:6813","nonce":35153}]},"public_addr":"172.21.2.120:6809/35153","cluster_addr":"172.21.2.120:6811/35153","heartbeat_back_addr":"172.21.2.120:6815/35153","heartbeat_front_addr":"172.21.2.120:6813/35153","state":["exists","up"]},{"osd":2,"uuid":"4e4493e3-f491-477d-bf09-7dc3a460ec4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":35154},{"type":"v1","addr":"172.21.2.120:6817","nonce":35154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6818","nonce":35154},{"type":"v1","addr":"172.21.2.120:6819","nonce":35154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6826","nonce":35154},{"type":"v1","addr":"172.21.2.120:6828","nonce":35154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":35154},{"type":"v1","addr":"172.21.2.120:6824","nonce":35154}]},"public_addr":"172.21.2.120:6817/35154","cluster_addr":"172.21.2.120:6819/35154","heartbeat_back_addr":"172.21.2.120:6828/35154","heartbeat_front_addr":"172.21.2.120:6824/35154","state":["exists","up"]},{"osd":3,"uuid":"d641d703-a808-437c-ac2d-db9d508cb1e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":103,"up_thru":200,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":53119},{"type":"v1","addr":"172.21.2.120:6822","nonce":53119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6823","nonce":53119},{"type":"v1","addr":"172.21.2.120:6825","nonce":53119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":53119},{"type":"v1","addr":"172.21.2.120:6831","nonce":53119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6827","nonce":53119},{"type":"v1","addr":"172.21.2.120:6829","nonce":53119}]},"public_addr":"172.21.2.120:6822/53119","cluster_addr":"172.21.2.120:6825/53119","heartbeat_back_addr":"172.21.2.120:6831/53119","heartbeat_front_addr":"172.21.2.120:6829/53119","state":["exists","up"]},{"osd":4,"uuid":"f2ba6f4b-e144-4b2c-9100-5d8c6e6e2680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6800","nonce":35127},{"type":"v1","addr":"172.21.2.136:6801","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6802","nonce":35127},{"type":"v1","addr":"172.21.2.136:6803","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6806","nonce":35127},{"type":"v1","addr":"172.21.2.136:6807","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6804","nonce":35127},{"type":"v1","addr":"172.21.2.136:6805","nonce":35127}]},"public_addr":"172.21.2.136:6801/35127","cluster_addr":"172.21.2.136:6803/35127","heartbeat_back_addr":"172.21.2.136:6807/35127","heartbeat_front_addr":"172.21.2.136:6805/35127","state":["exists","up"]},{"osd":5,"uuid":"04e6b644-75ed-4a4b-bad0-c3c6ec60db04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6808","nonce":35128},{"type":"v1","addr":"172.21.2.136:6809","nonce":35128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6810","nonce":35128},{"type":"v1","addr":"172.21.2.136:6811","nonce":35128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6814","nonce":35128},{"type":"v1","addr":"172.21.2.136:6815","nonce":35128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6812","nonce":35128},{"type":"v1","addr":"172.21.2.136:6813","nonce":35128}]},"public_addr":"172.21.2.136:6809/35128","cluster_addr":"172.21.2.136:6811/35128","heartbeat_back_addr":"172.21.2.136:6815/35128","heartbeat_front_addr":"172.21.2.136:6813/35128","state":["exists","up"]},{"osd":6,"uuid":"1bef0f21-9f6a-4580-8618-34b3c99bdd16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6816","nonce":35129},{"type":"v1","addr":"172.21.2.136:6817","nonce":35129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6818","nonce":35129},{"type":"v1","addr":"172.21.2.136:6819","nonce":35129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6823","nonce":35129},{"type":"v1","addr":"172.21.2.136:6825","nonce":35129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6820","nonce":35129},{"type":"v1","addr":"172.21.2.136:6821","nonce":35129}]},"public_addr":"172.21.2.136:6817/35129","cluster_addr":"172.21.2.136:6819/35129","heartbeat_back_addr":"172.21.2.136:6825/35129","heartbeat_front_addr":"172.21.2.136:6821/35129","state":["exists","up"]},{"osd":7,"uuid":"938c8347-9c90-48f7-bef9-fdb6aad88bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6822","nonce":35130},{"type":"v1","addr":"172.21.2.136:6824","nonce":35130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6826","nonce":35130},{"type":"v1","addr":"172.21.2.136:6827","nonce":35130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6830","nonce":35130},{"type":"v1","addr":"172.21.2.136:6831","nonce":35130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6828","nonce":35130},{"type":"v1","addr":"172.21.2.136:6829","nonce":35130}]},"public_addr":"172.21.2.136:6824/35130","cluster_addr":"172.21.2.136:6827/35130","heartbeat_back_addr":"172.21.2.136:6831/35130","heartbeat_front_addr":"172.21.2.136:6829/35130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.721854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:32.685787+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.339122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:09.743044+0000","dead_epoch":18},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.661501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.717093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:34.439797+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.618674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"2.21","osds":[6,7]}],"pg_upmap_items":[{"pgid":"1.8","mappings":[{"from":5,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1d","mappings":[{"from":4,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"isaprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"2","m":"1","name":"isaprofile","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:43:08.331 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats

Ignoring malparsed backtrace: 2021-03-11T17:43:16.547 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4
, instructed osd(s) 4 to deep-scrub
, 2021-03-11T17:43:16.771 INFO:tasks.ceph:Scrubbing osd.5

Ignoring malparsed backtrace: 2021-03-11T17:43:18.192 INFO:tasks.ceph:pgid 1.17 last_scrub_stamp 2021-03-11T17:42:44.288800+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.193 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2021-03-11T17:42:44.490732+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.194 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2021-03-11T17:43:06.994647+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=6, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:43:58.828 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2021-03-11T17:41:29.916342+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=41, tm_sec=29, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.829 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2021-03-11T17:42:22.136291+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.830 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-11T17:42:22.618589+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:44:20.744 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6
, 2021-03-11T17:44:20.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba036.front.sepia.ceph.com
, 2021-03-11T17:44:20.842 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7

Ignoring malparsed backtrace: 2021-03-11T17:44:40.229 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'
, 2021-03-11T17:44:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2021-03-11T17:44:40.252 DEBUG:teuthology.orchestra.run.gibba036:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7368837636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f7368837850]

Ignoring malparsed backtrace: 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:45.231 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.678589+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.292 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 4.2830000000000003e-06
, 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.345 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00043970499999999998
, 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.393 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:        {
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDECSubOpWrite(3.1ds0 158/157 ECSubWrite(tid=583, reqid=osd.5.0:583, at_version=158'102, trim_to=0'0, roll_forward_to=158'100))",
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:42:05.626282+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.459 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.049485+0000",
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0079951829999999995
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.021484904999999999,
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:28.529859+0000",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00020556799999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.529752+0000",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7636999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.617726+0000",
, 2021-03-11T17:42:45.563 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.089910559000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:45.591 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.666220+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00076042099999999995
, 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.665 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "started",
, 2021-03-11T17:42:46.051 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:12.547443+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                "client_info": {
, 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client": "client.4403",
, 2021-03-11T17:42:46.135 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.187 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.054735736
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.804747+0000",
, 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.37268568200000002
, 2021-03-11T17:42:46.276 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "initiated",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.524942+0000",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.525024+0000",
, 2021-03-11T17:42:46.332 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0031246450000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:37.455666+0000",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.91714987000000003

Ignoring malparsed backtrace: 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:46.403 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.543441+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.461 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 5.9699999999999996e-06
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace: 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.932 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:40:22.522708+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.9399999999999996e-06
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0092418210000000008
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:33.540464+0000",
, 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.5289999999999997e-06
, 2021-03-11T17:42:47.136 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.064631+0000",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.279816+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.265 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "description": "MOSDECSubOpRead(3.bs1 186/157 ECSubRead(tid=607, to_read={3:d64931e1:::gibba02043757-32:head=0,1331200,0}, subchunks={3:d64931e1:::gibba02043757-32:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:33.546160+0000",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 13.335403349,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.22042878499999999
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.672 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 1.8488999999999999e-05
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:45.527339+0000",
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.787 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:54.668498+0000",
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0052109050000000001
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1650
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.625151+0000",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.3100000000000006e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:06.092621+0000",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.9757000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:21.632581+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:21.632690+0000",
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 25.993322320000001,
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 3.888350532,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.965 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00022072899999999999
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.328 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 4.2470000000000002e-06
, 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.351 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.372 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626293+0000",
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.7078e-05
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.400 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626407+0000",
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.5360000000000001e-06
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.652389+0000",
, 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.027341317
, 2021-03-11T17:42:48.425 INFO:teuthology.orchestra.run.gibba020.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.672256+0000",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.024543906000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.704777+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.492 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00044892499999999998
, 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.517 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:41.094589+0000",
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.000145088
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.013825456999999999,
, 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:48.561 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.893503+0000",
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.918 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.178077017,
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.717149+0000",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.176429103

Ignoring malparsed backtrace: 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:48.969 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.742446+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.993 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:            }
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.023 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.074885183999999994
, 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.050 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:27.770791+0000",
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.100 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:28.102429+0000",
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0010558290000000001
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:59.771774+0000",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.7230000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.156 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.509165+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.513 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.153071+0000",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00064247600000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.898208932,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.096293769000000001,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.702543+0000",
, 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.795110295,
, 2021-03-11T17:42:49.563 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.111779344,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.596 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDPGPushReply(3.4s0 86/85 [PushReplyOp(3:2cd859fe:::gibba02043757-45:2e)])",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.819625+0000",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.678027782,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.2030000000000003e-05
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.649 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:12.380655+0000",
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 6.3300000000000004e-06
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.678 INFO:teuthology.orchestra.run.gibba036.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1475
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.703 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "all_read",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620324+0000",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.1453e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620503+0000",
, 2021-03-11T17:42:49.730 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0015644330000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:10.628567+0000",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0061397810000000004

Ignoring malparsed backtrace: 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.489268+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.797 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:51.965903+0000",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0032776649999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:52.741147+0000",
, 2021-03-11T17:42:50.159 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.77191896699999996

Ignoring malparsed backtrace: 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:28.615358+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.031812119999999999
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1686
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                },
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.625063+0000",
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.5679000000000002e-05
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.627796+0000",
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.351e-05
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.357 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.054156+0000",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9321e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "initiated",
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526387+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting for peered",
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526493+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.427 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.676370+0000",
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.002219431
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:36.963203+0000",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7416000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.648660+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.655900+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.860 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.882 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.930 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.075e-06
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.7482e-05
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.653925+0000",
, 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9043999999999999e-05
, 2021-03-11T17:42:50.980 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.724300+0000",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.8502e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:40.699325+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.156 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:{
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:    "ops": [],

Ignoring malparsed backtrace: 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1080
, 2021-03-11T17:42:51.383 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.529651+0000",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 7.165e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:51.429 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.474 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.452e-06
, 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.497 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.518 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.001181545
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.542310+0000",
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4488e-05
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.563 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.59616179700000005,
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.585 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:46.514706+0000",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4453000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.607 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.627664+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.629 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:06.093788+0000",
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 45.263869956000001,
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.255713739,

Ignoring malparsed backtrace: 2021-03-11T17:43:08.320 INFO:teuthology.orchestra.run.gibba020.stdout:
, 2021-03-11T17:43:08.321 INFO:teuthology.orchestra.run.gibba020.stdout:{"epoch":220,"fsid":"060c3692-b38e-46ba-a56d-20dacac6db00","created":"2021-03-11T17:39:24.709482+0000","modified":"2021-03-11T17:43:07.852798+0000","last_up_change":"2021-03-11T17:41:10.517652+0000","last_in_change":"2021-03-11T17:42:44.555830+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:39:33.822798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"119","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:39:35.121814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":38,"pg_placement_num":38,"pg_placement_num_target":38,"pg_num_target":38,"pg_num_pending":38,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"137","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:39:46.860506+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"157","auid":0,"snap_mode":"selfmanaged","snap_seq":168,"snap_epoch":198,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"isaprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5fb4eb39-9240-4477-a421-f0fc71779ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6800","nonce":35152},{"type":"v1","addr":"172.21.2.120:6801","nonce":35152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":35152},{"type":"v1","addr":"172.21.2.120:6803","nonce":35152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6806","nonce":35152},{"type":"v1","addr":"172.21.2.120:6807","nonce":35152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":35152},{"type":"v1","addr":"172.21.2.120:6805","nonce":35152}]},"public_addr":"172.21.2.120:6801/35152","cluster_addr":"172.21.2.120:6803/35152","heartbeat_back_addr":"172.21.2.120:6807/35152","heartbeat_front_addr":"172.21.2.120:6805/35152","state":["exists","up"]},{"osd":1,"uuid":"2d2ff228-8155-47d4-91bb-79f0ed2f526f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":35153},{"type":"v1","addr":"172.21.2.120:6809","nonce":35153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":35153},{"type":"v1","addr":"172.21.2.120:6811","nonce":35153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":35153},{"type":"v1","addr":"172.21.2.120:6815","nonce":35153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6812","nonce":35153},{"type":"v1","addr":"172.21.2.120:6813","nonce":35153}]},"public_addr":"172.21.2.120:6809/35153","cluster_addr":"172.21.2.120:6811/35153","heartbeat_back_addr":"172.21.2.120:6815/35153","heartbeat_front_addr":"172.21.2.120:6813/35153","state":["exists","up"]},{"osd":2,"uuid":"4e4493e3-f491-477d-bf09-7dc3a460ec4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":35154},{"type":"v1","addr":"172.21.2.120:6817","nonce":35154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6818","nonce":35154},{"type":"v1","addr":"172.21.2.120:6819","nonce":35154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6826","nonce":35154},{"type":"v1","addr":"172.21.2.120:6828","nonce":35154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":35154},{"type":"v1","addr":"172.21.2.120:6824","nonce":35154}]},"public_addr":"172.21.2.120:6817/35154","cluster_addr":"172.21.2.120:6819/35154","heartbeat_back_addr":"172.21.2.120:6828/35154","heartbeat_front_addr":"172.21.2.120:6824/35154","state":["exists","up"]},{"osd":3,"uuid":"d641d703-a808-437c-ac2d-db9d508cb1e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":103,"up_thru":200,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":53119},{"type":"v1","addr":"172.21.2.120:6822","nonce":53119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6823","nonce":53119},{"type":"v1","addr":"172.21.2.120:6825","nonce":53119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":53119},{"type":"v1","addr":"172.21.2.120:6831","nonce":53119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6827","nonce":53119},{"type":"v1","addr":"172.21.2.120:6829","nonce":53119}]},"public_addr":"172.21.2.120:6822/53119","cluster_addr":"172.21.2.120:6825/53119","heartbeat_back_addr":"172.21.2.120:6831/53119","heartbeat_front_addr":"172.21.2.120:6829/53119","state":["exists","up"]},{"osd":4,"uuid":"f2ba6f4b-e144-4b2c-9100-5d8c6e6e2680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6800","nonce":35127},{"type":"v1","addr":"172.21.2.136:6801","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6802","nonce":35127},{"type":"v1","addr":"172.21.2.136:6803","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6806","nonce":35127},{"type":"v1","addr":"172.21.2.136:6807","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6804","nonce":35127},{"type":"v1","addr":"172.21.2.136:6805","nonce":35127}]},"public_addr":"172.21.2.136:6801/35127","cluster_addr":"172.21.2.136:6803/35127","heartbeat_back_addr":"172.21.2.136:6807/35127","heartbeat_front_addr":"172.21.2.136:6805/35127","state":["exists","up"]},{"osd":5,"uuid":"04e6b644-75ed-4a4b-bad0-c3c6ec60db04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6808","nonce":35128},{"type":"v1","addr":"172.21.2.136:6809","nonce":35128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6810","nonce":35128},{"type":"v1","addr":"172.21.2.136:6811","nonce":35128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6814","nonce":35128},{"type":"v1","addr":"172.21.2.136:6815","nonce":35128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6812","nonce":35128},{"type":"v1","addr":"172.21.2.136:6813","nonce":35128}]},"public_addr":"172.21.2.136:6809/35128","cluster_addr":"172.21.2.136:6811/35128","heartbeat_back_addr":"172.21.2.136:6815/35128","heartbeat_front_addr":"172.21.2.136:6813/35128","state":["exists","up"]},{"osd":6,"uuid":"1bef0f21-9f6a-4580-8618-34b3c99bdd16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6816","nonce":35129},{"type":"v1","addr":"172.21.2.136:6817","nonce":35129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6818","nonce":35129},{"type":"v1","addr":"172.21.2.136:6819","nonce":35129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6823","nonce":35129},{"type":"v1","addr":"172.21.2.136:6825","nonce":35129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6820","nonce":35129},{"type":"v1","addr":"172.21.2.136:6821","nonce":35129}]},"public_addr":"172.21.2.136:6817/35129","cluster_addr":"172.21.2.136:6819/35129","heartbeat_back_addr":"172.21.2.136:6825/35129","heartbeat_front_addr":"172.21.2.136:6821/35129","state":["exists","up"]},{"osd":7,"uuid":"938c8347-9c90-48f7-bef9-fdb6aad88bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6822","nonce":35130},{"type":"v1","addr":"172.21.2.136:6824","nonce":35130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6826","nonce":35130},{"type":"v1","addr":"172.21.2.136:6827","nonce":35130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6830","nonce":35130},{"type":"v1","addr":"172.21.2.136:6831","nonce":35130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6828","nonce":35130},{"type":"v1","addr":"172.21.2.136:6829","nonce":35130}]},"public_addr":"172.21.2.136:6824/35130","cluster_addr":"172.21.2.136:6827/35130","heartbeat_back_addr":"172.21.2.136:6831/35130","heartbeat_front_addr":"172.21.2.136:6829/35130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.721854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:32.685787+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.339122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:09.743044+0000","dead_epoch":18},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.661501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.717093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:34.439797+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.618674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"2.21","osds":[6,7]}],"pg_upmap_items":[{"pgid":"1.8","mappings":[{"from":5,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1d","mappings":[{"from":4,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"isaprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"2","m":"1","name":"isaprofile","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:43:08.331 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats

Ignoring malparsed backtrace: 2021-03-11T17:43:16.547 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4
, instructed osd(s) 4 to deep-scrub
, 2021-03-11T17:43:16.771 INFO:tasks.ceph:Scrubbing osd.5

Ignoring malparsed backtrace: 2021-03-11T17:43:18.192 INFO:tasks.ceph:pgid 1.17 last_scrub_stamp 2021-03-11T17:42:44.288800+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.193 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2021-03-11T17:42:44.490732+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.194 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2021-03-11T17:43:06.994647+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=6, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:43:58.828 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2021-03-11T17:41:29.916342+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=41, tm_sec=29, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.829 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2021-03-11T17:42:22.136291+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.830 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-11T17:42:22.618589+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:44:20.744 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6
, 2021-03-11T17:44:20.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba036.front.sepia.ceph.com
, 2021-03-11T17:44:20.842 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7

Ignoring malparsed backtrace: 2021-03-11T17:44:40.229 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'
, 2021-03-11T17:44:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2021-03-11T17:44:40.252 DEBUG:teuthology.orchestra.run.gibba036:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7368837636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f7368837850]

Ignoring malparsed backtrace: 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:45.231 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.678589+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.292 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 4.2830000000000003e-06
, 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.345 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00043970499999999998
, 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.393 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:        {
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDECSubOpWrite(3.1ds0 158/157 ECSubWrite(tid=583, reqid=osd.5.0:583, at_version=158'102, trim_to=0'0, roll_forward_to=158'100))",
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:42:05.626282+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.459 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.049485+0000",
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0079951829999999995
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.021484904999999999,
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:28.529859+0000",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00020556799999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.529752+0000",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7636999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.617726+0000",
, 2021-03-11T17:42:45.563 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.089910559000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:45.591 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.666220+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00076042099999999995
, 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.665 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "started",
, 2021-03-11T17:42:46.051 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:12.547443+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                "client_info": {
, 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client": "client.4403",
, 2021-03-11T17:42:46.135 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.187 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.054735736
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.804747+0000",
, 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.37268568200000002
, 2021-03-11T17:42:46.276 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "initiated",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.524942+0000",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.525024+0000",
, 2021-03-11T17:42:46.332 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0031246450000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:37.455666+0000",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.91714987000000003

Ignoring malparsed backtrace: 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:46.403 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.543441+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.461 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 5.9699999999999996e-06
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace: 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.932 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:40:22.522708+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.9399999999999996e-06
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0092418210000000008
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:33.540464+0000",
, 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.5289999999999997e-06
, 2021-03-11T17:42:47.136 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.064631+0000",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.279816+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.265 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "description": "MOSDECSubOpRead(3.bs1 186/157 ECSubRead(tid=607, to_read={3:d64931e1:::gibba02043757-32:head=0,1331200,0}, subchunks={3:d64931e1:::gibba02043757-32:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:33.546160+0000",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 13.335403349,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.22042878499999999
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.672 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 1.8488999999999999e-05
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:45.527339+0000",
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.787 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:54.668498+0000",
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0052109050000000001
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1650
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.625151+0000",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.3100000000000006e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:06.092621+0000",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.9757000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:21.632581+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:21.632690+0000",
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 25.993322320000001,
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 3.888350532,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.965 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00022072899999999999
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.328 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 4.2470000000000002e-06
, 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.351 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.372 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626293+0000",
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.7078e-05
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.400 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626407+0000",
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.5360000000000001e-06
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.652389+0000",
, 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.027341317
, 2021-03-11T17:42:48.425 INFO:teuthology.orchestra.run.gibba020.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.672256+0000",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.024543906000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.704777+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.492 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00044892499999999998
, 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.517 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:41.094589+0000",
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.000145088
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.013825456999999999,
, 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:48.561 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.893503+0000",
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.918 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.178077017,
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.717149+0000",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.176429103

Ignoring malparsed backtrace: 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:48.969 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.742446+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.993 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:            }
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.023 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.074885183999999994
, 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.050 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:27.770791+0000",
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.100 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:28.102429+0000",
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0010558290000000001
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:59.771774+0000",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.7230000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.156 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.509165+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.513 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.153071+0000",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00064247600000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.898208932,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.096293769000000001,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.702543+0000",
, 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.795110295,
, 2021-03-11T17:42:49.563 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.111779344,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.596 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDPGPushReply(3.4s0 86/85 [PushReplyOp(3:2cd859fe:::gibba02043757-45:2e)])",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.819625+0000",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.678027782,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.2030000000000003e-05
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.649 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:12.380655+0000",
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 6.3300000000000004e-06
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.678 INFO:teuthology.orchestra.run.gibba036.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1475
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.703 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "all_read",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620324+0000",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.1453e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620503+0000",
, 2021-03-11T17:42:49.730 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0015644330000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:10.628567+0000",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0061397810000000004

Ignoring malparsed backtrace: 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.489268+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.797 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:51.965903+0000",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0032776649999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:52.741147+0000",
, 2021-03-11T17:42:50.159 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.77191896699999996

Ignoring malparsed backtrace: 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:28.615358+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.031812119999999999
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1686
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                },
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.625063+0000",
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.5679000000000002e-05
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.627796+0000",
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.351e-05
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.357 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.054156+0000",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9321e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "initiated",
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526387+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting for peered",
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526493+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.427 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.676370+0000",
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.002219431
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:36.963203+0000",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7416000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.648660+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.655900+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.860 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.882 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.930 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.075e-06
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.7482e-05
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.653925+0000",
, 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9043999999999999e-05
, 2021-03-11T17:42:50.980 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.724300+0000",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.8502e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:40.699325+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.156 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:{
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:    "ops": [],

Ignoring malparsed backtrace: 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1080
, 2021-03-11T17:42:51.383 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.529651+0000",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 7.165e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:51.429 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.474 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.452e-06
, 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.497 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.518 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.001181545
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.542310+0000",
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4488e-05
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.563 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.59616179700000005,
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.585 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:46.514706+0000",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4453000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.607 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.627664+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.629 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:06.093788+0000",
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 45.263869956000001,
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.255713739,

Ignoring malparsed backtrace: 2021-03-11T17:43:08.320 INFO:teuthology.orchestra.run.gibba020.stdout:
, 2021-03-11T17:43:08.321 INFO:teuthology.orchestra.run.gibba020.stdout:{"epoch":220,"fsid":"060c3692-b38e-46ba-a56d-20dacac6db00","created":"2021-03-11T17:39:24.709482+0000","modified":"2021-03-11T17:43:07.852798+0000","last_up_change":"2021-03-11T17:41:10.517652+0000","last_in_change":"2021-03-11T17:42:44.555830+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:39:33.822798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"119","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:39:35.121814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":38,"pg_placement_num":38,"pg_placement_num_target":38,"pg_num_target":38,"pg_num_pending":38,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"137","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:39:46.860506+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"157","auid":0,"snap_mode":"selfmanaged","snap_seq":168,"snap_epoch":198,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"isaprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5fb4eb39-9240-4477-a421-f0fc71779ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6800","nonce":35152},{"type":"v1","addr":"172.21.2.120:6801","nonce":35152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":35152},{"type":"v1","addr":"172.21.2.120:6803","nonce":35152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6806","nonce":35152},{"type":"v1","addr":"172.21.2.120:6807","nonce":35152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":35152},{"type":"v1","addr":"172.21.2.120:6805","nonce":35152}]},"public_addr":"172.21.2.120:6801/35152","cluster_addr":"172.21.2.120:6803/35152","heartbeat_back_addr":"172.21.2.120:6807/35152","heartbeat_front_addr":"172.21.2.120:6805/35152","state":["exists","up"]},{"osd":1,"uuid":"2d2ff228-8155-47d4-91bb-79f0ed2f526f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":35153},{"type":"v1","addr":"172.21.2.120:6809","nonce":35153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":35153},{"type":"v1","addr":"172.21.2.120:6811","nonce":35153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":35153},{"type":"v1","addr":"172.21.2.120:6815","nonce":35153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6812","nonce":35153},{"type":"v1","addr":"172.21.2.120:6813","nonce":35153}]},"public_addr":"172.21.2.120:6809/35153","cluster_addr":"172.21.2.120:6811/35153","heartbeat_back_addr":"172.21.2.120:6815/35153","heartbeat_front_addr":"172.21.2.120:6813/35153","state":["exists","up"]},{"osd":2,"uuid":"4e4493e3-f491-477d-bf09-7dc3a460ec4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":35154},{"type":"v1","addr":"172.21.2.120:6817","nonce":35154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6818","nonce":35154},{"type":"v1","addr":"172.21.2.120:6819","nonce":35154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6826","nonce":35154},{"type":"v1","addr":"172.21.2.120:6828","nonce":35154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":35154},{"type":"v1","addr":"172.21.2.120:6824","nonce":35154}]},"public_addr":"172.21.2.120:6817/35154","cluster_addr":"172.21.2.120:6819/35154","heartbeat_back_addr":"172.21.2.120:6828/35154","heartbeat_front_addr":"172.21.2.120:6824/35154","state":["exists","up"]},{"osd":3,"uuid":"d641d703-a808-437c-ac2d-db9d508cb1e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":103,"up_thru":200,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":53119},{"type":"v1","addr":"172.21.2.120:6822","nonce":53119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6823","nonce":53119},{"type":"v1","addr":"172.21.2.120:6825","nonce":53119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":53119},{"type":"v1","addr":"172.21.2.120:6831","nonce":53119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6827","nonce":53119},{"type":"v1","addr":"172.21.2.120:6829","nonce":53119}]},"public_addr":"172.21.2.120:6822/53119","cluster_addr":"172.21.2.120:6825/53119","heartbeat_back_addr":"172.21.2.120:6831/53119","heartbeat_front_addr":"172.21.2.120:6829/53119","state":["exists","up"]},{"osd":4,"uuid":"f2ba6f4b-e144-4b2c-9100-5d8c6e6e2680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6800","nonce":35127},{"type":"v1","addr":"172.21.2.136:6801","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6802","nonce":35127},{"type":"v1","addr":"172.21.2.136:6803","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6806","nonce":35127},{"type":"v1","addr":"172.21.2.136:6807","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6804","nonce":35127},{"type":"v1","addr":"172.21.2.136:6805","nonce":35127}]},"public_addr":"172.21.2.136:6801/35127","cluster_addr":"172.21.2.136:6803/35127","heartbeat_back_addr":"172.21.2.136:6807/35127","heartbeat_front_addr":"172.21.2.136:6805/35127","state":["exists","up"]},{"osd":5,"uuid":"04e6b644-75ed-4a4b-bad0-c3c6ec60db04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6808","nonce":35128},{"type":"v1","addr":"172.21.2.136:6809","nonce":35128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6810","nonce":35128},{"type":"v1","addr":"172.21.2.136:6811","nonce":35128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6814","nonce":35128},{"type":"v1","addr":"172.21.2.136:6815","nonce":35128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6812","nonce":35128},{"type":"v1","addr":"172.21.2.136:6813","nonce":35128}]},"public_addr":"172.21.2.136:6809/35128","cluster_addr":"172.21.2.136:6811/35128","heartbeat_back_addr":"172.21.2.136:6815/35128","heartbeat_front_addr":"172.21.2.136:6813/35128","state":["exists","up"]},{"osd":6,"uuid":"1bef0f21-9f6a-4580-8618-34b3c99bdd16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6816","nonce":35129},{"type":"v1","addr":"172.21.2.136:6817","nonce":35129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6818","nonce":35129},{"type":"v1","addr":"172.21.2.136:6819","nonce":35129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6823","nonce":35129},{"type":"v1","addr":"172.21.2.136:6825","nonce":35129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6820","nonce":35129},{"type":"v1","addr":"172.21.2.136:6821","nonce":35129}]},"public_addr":"172.21.2.136:6817/35129","cluster_addr":"172.21.2.136:6819/35129","heartbeat_back_addr":"172.21.2.136:6825/35129","heartbeat_front_addr":"172.21.2.136:6821/35129","state":["exists","up"]},{"osd":7,"uuid":"938c8347-9c90-48f7-bef9-fdb6aad88bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6822","nonce":35130},{"type":"v1","addr":"172.21.2.136:6824","nonce":35130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6826","nonce":35130},{"type":"v1","addr":"172.21.2.136:6827","nonce":35130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6830","nonce":35130},{"type":"v1","addr":"172.21.2.136:6831","nonce":35130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6828","nonce":35130},{"type":"v1","addr":"172.21.2.136:6829","nonce":35130}]},"public_addr":"172.21.2.136:6824/35130","cluster_addr":"172.21.2.136:6827/35130","heartbeat_back_addr":"172.21.2.136:6831/35130","heartbeat_front_addr":"172.21.2.136:6829/35130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.721854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:32.685787+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.339122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:09.743044+0000","dead_epoch":18},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.661501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.717093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:34.439797+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.618674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"2.21","osds":[6,7]}],"pg_upmap_items":[{"pgid":"1.8","mappings":[{"from":5,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1d","mappings":[{"from":4,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"isaprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"2","m":"1","name":"isaprofile","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:43:08.331 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats

Ignoring malparsed backtrace: 2021-03-11T17:43:16.547 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4
, instructed osd(s) 4 to deep-scrub
, 2021-03-11T17:43:16.771 INFO:tasks.ceph:Scrubbing osd.5

Ignoring malparsed backtrace: 2021-03-11T17:43:18.192 INFO:tasks.ceph:pgid 1.17 last_scrub_stamp 2021-03-11T17:42:44.288800+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.193 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2021-03-11T17:42:44.490732+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.194 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2021-03-11T17:43:06.994647+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=6, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:43:58.828 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2021-03-11T17:41:29.916342+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=41, tm_sec=29, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.829 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2021-03-11T17:42:22.136291+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.830 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-11T17:42:22.618589+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:44:20.744 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6
, 2021-03-11T17:44:20.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba036.front.sepia.ceph.com
, 2021-03-11T17:44:20.842 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7

Ignoring malparsed backtrace: 2021-03-11T17:44:40.229 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'
, 2021-03-11T17:44:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2021-03-11T17:44:40.252 DEBUG:teuthology.orchestra.run.gibba036:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7368837636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f7368837850]

Ignoring malparsed backtrace: 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:45.231 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.678589+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.292 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 4.2830000000000003e-06
, 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.345 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00043970499999999998
, 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.393 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:        {
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDECSubOpWrite(3.1ds0 158/157 ECSubWrite(tid=583, reqid=osd.5.0:583, at_version=158'102, trim_to=0'0, roll_forward_to=158'100))",
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:42:05.626282+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.459 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.049485+0000",
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0079951829999999995
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.021484904999999999,
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:28.529859+0000",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00020556799999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.529752+0000",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7636999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.617726+0000",
, 2021-03-11T17:42:45.563 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.089910559000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:45.591 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.666220+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00076042099999999995
, 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.665 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "started",
, 2021-03-11T17:42:46.051 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:12.547443+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                "client_info": {
, 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client": "client.4403",
, 2021-03-11T17:42:46.135 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.187 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.054735736
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.804747+0000",
, 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.37268568200000002
, 2021-03-11T17:42:46.276 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "initiated",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.524942+0000",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.525024+0000",
, 2021-03-11T17:42:46.332 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0031246450000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:37.455666+0000",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.91714987000000003

Ignoring malparsed backtrace: 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:46.403 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.543441+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.461 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 5.9699999999999996e-06
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace: 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.932 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:40:22.522708+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.9399999999999996e-06
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0092418210000000008
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:33.540464+0000",
, 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.5289999999999997e-06
, 2021-03-11T17:42:47.136 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.064631+0000",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.279816+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.265 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "description": "MOSDECSubOpRead(3.bs1 186/157 ECSubRead(tid=607, to_read={3:d64931e1:::gibba02043757-32:head=0,1331200,0}, subchunks={3:d64931e1:::gibba02043757-32:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:33.546160+0000",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 13.335403349,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.22042878499999999
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.672 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 1.8488999999999999e-05
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:45.527339+0000",
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.787 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:54.668498+0000",
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0052109050000000001
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1650
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.625151+0000",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.3100000000000006e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:06.092621+0000",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.9757000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:21.632581+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:21.632690+0000",
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 25.993322320000001,
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 3.888350532,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.965 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00022072899999999999
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.328 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 4.2470000000000002e-06
, 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.351 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.372 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626293+0000",
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.7078e-05
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.400 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626407+0000",
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.5360000000000001e-06
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.652389+0000",
, 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.027341317
, 2021-03-11T17:42:48.425 INFO:teuthology.orchestra.run.gibba020.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.672256+0000",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.024543906000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.704777+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.492 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00044892499999999998
, 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.517 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:41.094589+0000",
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.000145088
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.013825456999999999,
, 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:48.561 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.893503+0000",
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.918 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.178077017,
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.717149+0000",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.176429103

Ignoring malparsed backtrace: 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:48.969 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.742446+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.993 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:            }
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.023 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.074885183999999994
, 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.050 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:27.770791+0000",
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.100 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:28.102429+0000",
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0010558290000000001
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:59.771774+0000",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.7230000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.156 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.509165+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.513 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.153071+0000",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00064247600000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.898208932,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.096293769000000001,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.702543+0000",
, 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.795110295,
, 2021-03-11T17:42:49.563 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.111779344,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.596 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDPGPushReply(3.4s0 86/85 [PushReplyOp(3:2cd859fe:::gibba02043757-45:2e)])",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.819625+0000",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.678027782,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.2030000000000003e-05
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.649 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:12.380655+0000",
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 6.3300000000000004e-06
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.678 INFO:teuthology.orchestra.run.gibba036.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1475
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.703 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "all_read",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620324+0000",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.1453e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620503+0000",
, 2021-03-11T17:42:49.730 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0015644330000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:10.628567+0000",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0061397810000000004

Ignoring malparsed backtrace: 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.489268+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.797 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:51.965903+0000",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0032776649999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:52.741147+0000",
, 2021-03-11T17:42:50.159 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.77191896699999996

Ignoring malparsed backtrace: 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:28.615358+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.031812119999999999
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1686
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                },
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.625063+0000",
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.5679000000000002e-05
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.627796+0000",
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.351e-05
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.357 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.054156+0000",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9321e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "initiated",
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526387+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting for peered",
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526493+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.427 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.676370+0000",
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.002219431
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:36.963203+0000",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7416000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.648660+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.655900+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.860 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.882 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.930 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.075e-06
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.7482e-05
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.653925+0000",
, 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9043999999999999e-05
, 2021-03-11T17:42:50.980 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.724300+0000",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.8502e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:40.699325+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.156 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:{
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:    "ops": [],

Ignoring malparsed backtrace: 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1080
, 2021-03-11T17:42:51.383 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.529651+0000",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 7.165e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:51.429 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.474 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.452e-06
, 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.497 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.518 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.001181545
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.542310+0000",
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4488e-05
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.563 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.59616179700000005,
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.585 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:46.514706+0000",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4453000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.607 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.627664+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.629 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:06.093788+0000",
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 45.263869956000001,
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.255713739,

Ignoring malparsed backtrace: 2021-03-11T17:43:08.320 INFO:teuthology.orchestra.run.gibba020.stdout:
, 2021-03-11T17:43:08.321 INFO:teuthology.orchestra.run.gibba020.stdout:{"epoch":220,"fsid":"060c3692-b38e-46ba-a56d-20dacac6db00","created":"2021-03-11T17:39:24.709482+0000","modified":"2021-03-11T17:43:07.852798+0000","last_up_change":"2021-03-11T17:41:10.517652+0000","last_in_change":"2021-03-11T17:42:44.555830+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:39:33.822798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"119","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:39:35.121814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":38,"pg_placement_num":38,"pg_placement_num_target":38,"pg_num_target":38,"pg_num_pending":38,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"137","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:39:46.860506+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"157","auid":0,"snap_mode":"selfmanaged","snap_seq":168,"snap_epoch":198,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"isaprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5fb4eb39-9240-4477-a421-f0fc71779ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6800","nonce":35152},{"type":"v1","addr":"172.21.2.120:6801","nonce":35152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":35152},{"type":"v1","addr":"172.21.2.120:6803","nonce":35152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6806","nonce":35152},{"type":"v1","addr":"172.21.2.120:6807","nonce":35152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":35152},{"type":"v1","addr":"172.21.2.120:6805","nonce":35152}]},"public_addr":"172.21.2.120:6801/35152","cluster_addr":"172.21.2.120:6803/35152","heartbeat_back_addr":"172.21.2.120:6807/35152","heartbeat_front_addr":"172.21.2.120:6805/35152","state":["exists","up"]},{"osd":1,"uuid":"2d2ff228-8155-47d4-91bb-79f0ed2f526f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":35153},{"type":"v1","addr":"172.21.2.120:6809","nonce":35153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":35153},{"type":"v1","addr":"172.21.2.120:6811","nonce":35153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":35153},{"type":"v1","addr":"172.21.2.120:6815","nonce":35153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6812","nonce":35153},{"type":"v1","addr":"172.21.2.120:6813","nonce":35153}]},"public_addr":"172.21.2.120:6809/35153","cluster_addr":"172.21.2.120:6811/35153","heartbeat_back_addr":"172.21.2.120:6815/35153","heartbeat_front_addr":"172.21.2.120:6813/35153","state":["exists","up"]},{"osd":2,"uuid":"4e4493e3-f491-477d-bf09-7dc3a460ec4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":35154},{"type":"v1","addr":"172.21.2.120:6817","nonce":35154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6818","nonce":35154},{"type":"v1","addr":"172.21.2.120:6819","nonce":35154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6826","nonce":35154},{"type":"v1","addr":"172.21.2.120:6828","nonce":35154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":35154},{"type":"v1","addr":"172.21.2.120:6824","nonce":35154}]},"public_addr":"172.21.2.120:6817/35154","cluster_addr":"172.21.2.120:6819/35154","heartbeat_back_addr":"172.21.2.120:6828/35154","heartbeat_front_addr":"172.21.2.120:6824/35154","state":["exists","up"]},{"osd":3,"uuid":"d641d703-a808-437c-ac2d-db9d508cb1e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":103,"up_thru":200,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":53119},{"type":"v1","addr":"172.21.2.120:6822","nonce":53119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6823","nonce":53119},{"type":"v1","addr":"172.21.2.120:6825","nonce":53119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":53119},{"type":"v1","addr":"172.21.2.120:6831","nonce":53119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6827","nonce":53119},{"type":"v1","addr":"172.21.2.120:6829","nonce":53119}]},"public_addr":"172.21.2.120:6822/53119","cluster_addr":"172.21.2.120:6825/53119","heartbeat_back_addr":"172.21.2.120:6831/53119","heartbeat_front_addr":"172.21.2.120:6829/53119","state":["exists","up"]},{"osd":4,"uuid":"f2ba6f4b-e144-4b2c-9100-5d8c6e6e2680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6800","nonce":35127},{"type":"v1","addr":"172.21.2.136:6801","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6802","nonce":35127},{"type":"v1","addr":"172.21.2.136:6803","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6806","nonce":35127},{"type":"v1","addr":"172.21.2.136:6807","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6804","nonce":35127},{"type":"v1","addr":"172.21.2.136:6805","nonce":35127}]},"public_addr":"172.21.2.136:6801/35127","cluster_addr":"172.21.2.136:6803/35127","heartbeat_back_addr":"172.21.2.136:6807/35127","heartbeat_front_addr":"172.21.2.136:6805/35127","state":["exists","up"]},{"osd":5,"uuid":"04e6b644-75ed-4a4b-bad0-c3c6ec60db04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6808","nonce":35128},{"type":"v1","addr":"172.21.2.136:6809","nonce":35128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6810","nonce":35128},{"type":"v1","addr":"172.21.2.136:6811","nonce":35128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6814","nonce":35128},{"type":"v1","addr":"172.21.2.136:6815","nonce":35128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6812","nonce":35128},{"type":"v1","addr":"172.21.2.136:6813","nonce":35128}]},"public_addr":"172.21.2.136:6809/35128","cluster_addr":"172.21.2.136:6811/35128","heartbeat_back_addr":"172.21.2.136:6815/35128","heartbeat_front_addr":"172.21.2.136:6813/35128","state":["exists","up"]},{"osd":6,"uuid":"1bef0f21-9f6a-4580-8618-34b3c99bdd16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6816","nonce":35129},{"type":"v1","addr":"172.21.2.136:6817","nonce":35129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6818","nonce":35129},{"type":"v1","addr":"172.21.2.136:6819","nonce":35129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6823","nonce":35129},{"type":"v1","addr":"172.21.2.136:6825","nonce":35129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6820","nonce":35129},{"type":"v1","addr":"172.21.2.136:6821","nonce":35129}]},"public_addr":"172.21.2.136:6817/35129","cluster_addr":"172.21.2.136:6819/35129","heartbeat_back_addr":"172.21.2.136:6825/35129","heartbeat_front_addr":"172.21.2.136:6821/35129","state":["exists","up"]},{"osd":7,"uuid":"938c8347-9c90-48f7-bef9-fdb6aad88bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6822","nonce":35130},{"type":"v1","addr":"172.21.2.136:6824","nonce":35130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6826","nonce":35130},{"type":"v1","addr":"172.21.2.136:6827","nonce":35130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6830","nonce":35130},{"type":"v1","addr":"172.21.2.136:6831","nonce":35130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6828","nonce":35130},{"type":"v1","addr":"172.21.2.136:6829","nonce":35130}]},"public_addr":"172.21.2.136:6824/35130","cluster_addr":"172.21.2.136:6827/35130","heartbeat_back_addr":"172.21.2.136:6831/35130","heartbeat_front_addr":"172.21.2.136:6829/35130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.721854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:32.685787+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.339122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:09.743044+0000","dead_epoch":18},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.661501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.717093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:34.439797+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.618674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"2.21","osds":[6,7]}],"pg_upmap_items":[{"pgid":"1.8","mappings":[{"from":5,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1d","mappings":[{"from":4,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"isaprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"2","m":"1","name":"isaprofile","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:43:08.331 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats

Ignoring malparsed backtrace: 2021-03-11T17:43:16.547 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4
, instructed osd(s) 4 to deep-scrub
, 2021-03-11T17:43:16.771 INFO:tasks.ceph:Scrubbing osd.5

Ignoring malparsed backtrace: 2021-03-11T17:43:18.192 INFO:tasks.ceph:pgid 1.17 last_scrub_stamp 2021-03-11T17:42:44.288800+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.193 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2021-03-11T17:42:44.490732+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.194 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2021-03-11T17:43:06.994647+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=6, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:43:58.828 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2021-03-11T17:41:29.916342+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=41, tm_sec=29, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.829 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2021-03-11T17:42:22.136291+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.830 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-11T17:42:22.618589+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:44:20.744 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6
, 2021-03-11T17:44:20.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba036.front.sepia.ceph.com
, 2021-03-11T17:44:20.842 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7

Ignoring malparsed backtrace: 2021-03-11T17:44:40.229 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'
, 2021-03-11T17:44:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2021-03-11T17:44:40.252 DEBUG:teuthology.orchestra.run.gibba036:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f7368837636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f7368837850]

Ignoring malparsed backtrace: 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.230 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:45.231 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.678589+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.292 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.293 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 4.2830000000000003e-06
, 2021-03-11T17:42:45.344 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.345 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00043970499999999998
, 2021-03-11T17:42:45.392 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.393 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:        {
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDECSubOpWrite(3.1ds0 158/157 ECSubWrite(tid=583, reqid=osd.5.0:583, at_version=158'102, trim_to=0'0, roll_forward_to=158'100))",
, 2021-03-11T17:42:45.435 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:42:05.626282+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.459 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.049485+0000",
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0079951829999999995
, 2021-03-11T17:42:45.460 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.021484904999999999,
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:45.484 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:28.529859+0000",
, 2021-03-11T17:42:45.512 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00020556799999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.529752+0000",
, 2021-03-11T17:42:45.537 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7636999999999997e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.562 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.617726+0000",
, 2021-03-11T17:42:45.563 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.089910559000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:45.591 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:45.592 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.666220+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:45.632 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00076042099999999995
, 2021-03-11T17:42:45.664 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:45.665 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.050 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "started",
, 2021-03-11T17:42:46.051 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:12.547443+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                "client_info": {
, 2021-03-11T17:42:46.134 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client": "client.4403",
, 2021-03-11T17:42:46.135 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.187 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.188 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.054735736
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.235 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.804747+0000",
, 2021-03-11T17:42:46.275 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.37268568200000002
, 2021-03-11T17:42:46.276 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "initiated",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.524942+0000",
, 2021-03-11T17:42:46.309 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:46.331 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:22.525024+0000",
, 2021-03-11T17:42:46.332 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0031246450000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:37.455666+0000",
, 2021-03-11T17:42:46.376 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.91714987000000003

Ignoring malparsed backtrace: 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.402 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:46.403 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.543441+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.432 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.460 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.461 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 5.9699999999999996e-06
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace: 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:46.903 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:46.932 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:46.933 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:40:22.522708+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:46.959 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.001 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.9399999999999996e-06
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.033 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0092418210000000008
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.063 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:33.540464+0000",
, 2021-03-11T17:42:47.135 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 6.5289999999999997e-06
, 2021-03-11T17:42:47.136 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.064631+0000",
, 2021-03-11T17:42:47.180 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "header_read",
, 2021-03-11T17:42:47.231 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.279816+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.265 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.266 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "description": "MOSDECSubOpRead(3.bs1 186/157 ECSubRead(tid=607, to_read={3:d64931e1:::gibba02043757-32:head=0,1331200,0}, subchunks={3:d64931e1:::gibba02043757-32:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:33.546160+0000",
, 2021-03-11T17:42:47.289 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 13.335403349,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.22042878499999999
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.312 INFO:teuthology.orchestra.run.gibba020.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.671 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.672 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                    }
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:47.709 INFO:teuthology.orchestra.run.gibba020.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 1.8488999999999999e-05
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.740 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:45.527339+0000",
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0
, 2021-03-11T17:42:47.764 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.787 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:54.668498+0000",
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.0052109050000000001
, 2021-03-11T17:42:47.788 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1650
, 2021-03-11T17:42:47.812 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.625151+0000",
, 2021-03-11T17:42:47.833 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.3100000000000006e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:06.092621+0000",
, 2021-03-11T17:42:47.863 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 9.9757000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:47.890 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:21.632581+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:21.632690+0000",
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 25.993322320000001,
, 2021-03-11T17:42:47.914 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 3.888350532,

Ignoring malparsed backtrace: 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.941 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.965 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:47.966 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00022072899999999999
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:47.994 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.328 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.329 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 4.2470000000000002e-06
, 2021-03-11T17:42:48.350 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.351 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.372 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626293+0000",
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.7078e-05
, 2021-03-11T17:42:48.373 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.400 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:09.626407+0000",
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 3.5360000000000001e-06
, 2021-03-11T17:42:48.401 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.652389+0000",
, 2021-03-11T17:42:48.424 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.027341317
, 2021-03-11T17:42:48.425 INFO:teuthology.orchestra.run.gibba020.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.672256+0000",
, 2021-03-11T17:42:48.446 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.024543906000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",
, 2021-03-11T17:42:48.468 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:30.704777+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.491 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:48.492 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.00044892499999999998
, 2021-03-11T17:42:48.516 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:48.517 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:41.094589+0000",
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.000145088
, 2021-03-11T17:42:48.538 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.013825456999999999,
, 2021-03-11T17:42:48.560 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:48.561 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.893503+0000",
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:48.889 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:48.918 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.178077017,
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {
, 2021-03-11T17:42:48.919 INFO:teuthology.orchestra.run.gibba036.stdout:                "flag_point": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.717149+0000",
, 2021-03-11T17:42:48.945 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.176429103

Ignoring malparsed backtrace: 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:48.968 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:48.969 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:57.742446+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:48.993 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:            }
, 2021-03-11T17:42:48.994 INFO:teuthology.orchestra.run.gibba036.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.023 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.025 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.074885183999999994
, 2021-03-11T17:42:49.049 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.050 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:27.770791+0000",
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0
, 2021-03-11T17:42:49.075 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.100 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:28.102429+0000",
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0010558290000000001
, 2021-03-11T17:42:49.101 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:59.771774+0000",
, 2021-03-11T17:42:49.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.7230000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.155 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:49.156 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.509165+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.179 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.513 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:53.153071+0000",
, 2021-03-11T17:42:49.514 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.00064247600000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.898208932,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.096293769000000001,
, 2021-03-11T17:42:49.540 INFO:teuthology.orchestra.run.gibba036.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.702543+0000",
, 2021-03-11T17:42:49.562 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.795110295,
, 2021-03-11T17:42:49.563 INFO:teuthology.orchestra.run.gibba036.stdout:            "duration": 0.111779344,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.596 INFO:teuthology.orchestra.run.gibba036.stdout:            "description": "MOSDPGPushReply(3.4s0 86/85 [PushReplyOp(3:2cd859fe:::gibba02043757-45:2e)])",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "initiated_at": "2021-03-11T17:40:53.819625+0000",
, 2021-03-11T17:42:49.597 INFO:teuthology.orchestra.run.gibba036.stdout:            "age": 115.678027782,

Ignoring malparsed backtrace: 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.2030000000000003e-05
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.625 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:49.649 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:12.380655+0000",
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 6.3300000000000004e-06
, 2021-03-11T17:42:49.650 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.678 INFO:teuthology.orchestra.run.gibba036.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1475
, 2021-03-11T17:42:49.679 INFO:teuthology.orchestra.run.gibba036.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:49.703 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "all_read",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620324+0000",
, 2021-03-11T17:42:49.704 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.1453e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:42:49.729 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.620503+0000",
, 2021-03-11T17:42:49.730 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0015644330000000001

Ignoring malparsed backtrace: 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:10.628567+0000",
, 2021-03-11T17:42:49.753 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0061397810000000004

Ignoring malparsed backtrace: 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:49.775 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.489268+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:49.797 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:49.798 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:51.965903+0000",
, 2021-03-11T17:42:50.128 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.0032776649999999999

Ignoring malparsed backtrace: 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "done",
, 2021-03-11T17:42:50.158 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:39:52.741147+0000",
, 2021-03-11T17:42:50.159 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.77191896699999996

Ignoring malparsed backtrace: 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "dispatched",
, 2021-03-11T17:42:50.189 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:28.615358+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.213 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:                ]
, 2021-03-11T17:42:50.236 INFO:teuthology.orchestra.run.gibba036.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.031812119999999999
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                    }
, 2021-03-11T17:42:50.261 INFO:teuthology.orchestra.run.gibba036.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                    "tid": 1686
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                },
, 2021-03-11T17:42:50.288 INFO:teuthology.orchestra.run.gibba036.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.625063+0000",
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 3.5679000000000002e-05
, 2021-03-11T17:42:50.312 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:05.627796+0000",
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.351e-05
, 2021-03-11T17:42:50.336 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.357 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:09.054156+0000",
, 2021-03-11T17:42:50.358 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9321e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "initiated",
, 2021-03-11T17:42:50.379 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526387+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "waiting for peered",
, 2021-03-11T17:42:50.402 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:14.526493+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.427 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.428 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:40:58.676370+0000",
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 0.002219431
, 2021-03-11T17:42:50.765 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:36.963203+0000",
, 2021-03-11T17:42:50.790 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 5.7416000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "header_read",
, 2021-03-11T17:42:50.814 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.648660+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",
, 2021-03-11T17:42:50.836 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:41:54.655900+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.859 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.860 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.882 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.883 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "sub_op_started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:50.906 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-03-11T17:42:50.930 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 2.075e-06
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.931 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.7482e-05
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    },
, 2021-03-11T17:42:50.955 INFO:teuthology.orchestra.run.gibba036.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.653925+0000",
, 2021-03-11T17:42:50.979 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.9043999999999999e-05
, 2021-03-11T17:42:50.980 INFO:teuthology.orchestra.run.gibba036.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:30.724300+0000",
, 2021-03-11T17:42:51.003 INFO:teuthology.orchestra.run.gibba036.stdout:                        "duration": 1.8502e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                    {
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.029 INFO:teuthology.orchestra.run.gibba036.stdout:                        "time": "2021-03-11T17:42:40.699325+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.156 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:{
, 2021-03-11T17:42:51.254 INFO:teuthology.orchestra.run.gibba020.stdout:    "ops": [],

Ignoring malparsed backtrace: 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "client_addr": "172.21.2.120:0/3660656035",
, 2021-03-11T17:42:51.382 INFO:teuthology.orchestra.run.gibba020.stdout:                    "tid": 1080
, 2021-03-11T17:42:51.383 INFO:teuthology.orchestra.run.gibba020.stdout:                },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "all_read",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:11.529651+0000",
, 2021-03-11T17:42:51.406 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 7.165e-06

Ignoring malparsed backtrace: 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.428 INFO:teuthology.orchestra.run.gibba020.stdout:        },
, 2021-03-11T17:42:51.429 INFO:teuthology.orchestra.run.gibba020.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:                ]
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:            }
, 2021-03-11T17:42:51.451 INFO:teuthology.orchestra.run.gibba020.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.473 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.474 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "sub_op_committed",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.452e-06
, 2021-03-11T17:42:51.496 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.497 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.518 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 0.001181545
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    },
, 2021-03-11T17:42:51.519 INFO:teuthology.orchestra.run.gibba020.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:36.542310+0000",
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4488e-05
, 2021-03-11T17:42:51.541 INFO:teuthology.orchestra.run.gibba020.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:42:51.563 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.59616179700000005,
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:            "type_data": {
, 2021-03-11T17:42:51.564 INFO:teuthology.orchestra.run.gibba020.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.585 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "reached_pg",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:41:46.514706+0000",
, 2021-03-11T17:42:51.586 INFO:teuthology.orchestra.run.gibba020.stdout:                        "duration": 2.4453000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:42:51.607 INFO:teuthology.orchestra.run.gibba020.stdout:                    {
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "event": "throttled",
, 2021-03-11T17:42:51.608 INFO:teuthology.orchestra.run.gibba020.stdout:                        "time": "2021-03-11T17:42:05.627664+0000",

Ignoring malparsed backtrace: 2021-03-11T17:42:51.629 INFO:teuthology.orchestra.run.gibba020.stdout:            "initiated_at": "2021-03-11T17:42:06.093788+0000",
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "age": 45.263869956000001,
, 2021-03-11T17:42:51.630 INFO:teuthology.orchestra.run.gibba020.stdout:            "duration": 0.255713739,

Ignoring malparsed backtrace: 2021-03-11T17:43:08.320 INFO:teuthology.orchestra.run.gibba020.stdout:
, 2021-03-11T17:43:08.321 INFO:teuthology.orchestra.run.gibba020.stdout:{"epoch":220,"fsid":"060c3692-b38e-46ba-a56d-20dacac6db00","created":"2021-03-11T17:39:24.709482+0000","modified":"2021-03-11T17:43:07.852798+0000","last_up_change":"2021-03-11T17:41:10.517652+0000","last_in_change":"2021-03-11T17:42:44.555830+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T17:39:33.822798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"119","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T17:39:35.121814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":38,"pg_placement_num":38,"pg_placement_num_target":38,"pg_num_target":38,"pg_num_pending":38,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"137","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T17:39:46.860506+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"157","auid":0,"snap_mode":"selfmanaged","snap_seq":168,"snap_epoch":198,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"isaprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"5fb4eb39-9240-4477-a421-f0fc71779ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6800","nonce":35152},{"type":"v1","addr":"172.21.2.120:6801","nonce":35152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6802","nonce":35152},{"type":"v1","addr":"172.21.2.120:6803","nonce":35152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6806","nonce":35152},{"type":"v1","addr":"172.21.2.120:6807","nonce":35152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6804","nonce":35152},{"type":"v1","addr":"172.21.2.120:6805","nonce":35152}]},"public_addr":"172.21.2.120:6801/35152","cluster_addr":"172.21.2.120:6803/35152","heartbeat_back_addr":"172.21.2.120:6807/35152","heartbeat_front_addr":"172.21.2.120:6805/35152","state":["exists","up"]},{"osd":1,"uuid":"2d2ff228-8155-47d4-91bb-79f0ed2f526f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6808","nonce":35153},{"type":"v1","addr":"172.21.2.120:6809","nonce":35153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6810","nonce":35153},{"type":"v1","addr":"172.21.2.120:6811","nonce":35153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6814","nonce":35153},{"type":"v1","addr":"172.21.2.120:6815","nonce":35153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6812","nonce":35153},{"type":"v1","addr":"172.21.2.120:6813","nonce":35153}]},"public_addr":"172.21.2.120:6809/35153","cluster_addr":"172.21.2.120:6811/35153","heartbeat_back_addr":"172.21.2.120:6815/35153","heartbeat_front_addr":"172.21.2.120:6813/35153","state":["exists","up"]},{"osd":2,"uuid":"4e4493e3-f491-477d-bf09-7dc3a460ec4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6816","nonce":35154},{"type":"v1","addr":"172.21.2.120:6817","nonce":35154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6818","nonce":35154},{"type":"v1","addr":"172.21.2.120:6819","nonce":35154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6826","nonce":35154},{"type":"v1","addr":"172.21.2.120:6828","nonce":35154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6821","nonce":35154},{"type":"v1","addr":"172.21.2.120:6824","nonce":35154}]},"public_addr":"172.21.2.120:6817/35154","cluster_addr":"172.21.2.120:6819/35154","heartbeat_back_addr":"172.21.2.120:6828/35154","heartbeat_front_addr":"172.21.2.120:6824/35154","state":["exists","up"]},{"osd":3,"uuid":"d641d703-a808-437c-ac2d-db9d508cb1e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":103,"up_thru":200,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6820","nonce":53119},{"type":"v1","addr":"172.21.2.120:6822","nonce":53119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6823","nonce":53119},{"type":"v1","addr":"172.21.2.120:6825","nonce":53119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6830","nonce":53119},{"type":"v1","addr":"172.21.2.120:6831","nonce":53119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.120:6827","nonce":53119},{"type":"v1","addr":"172.21.2.120:6829","nonce":53119}]},"public_addr":"172.21.2.120:6822/53119","cluster_addr":"172.21.2.120:6825/53119","heartbeat_back_addr":"172.21.2.120:6831/53119","heartbeat_front_addr":"172.21.2.120:6829/53119","state":["exists","up"]},{"osd":4,"uuid":"f2ba6f4b-e144-4b2c-9100-5d8c6e6e2680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6800","nonce":35127},{"type":"v1","addr":"172.21.2.136:6801","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6802","nonce":35127},{"type":"v1","addr":"172.21.2.136:6803","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6806","nonce":35127},{"type":"v1","addr":"172.21.2.136:6807","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6804","nonce":35127},{"type":"v1","addr":"172.21.2.136:6805","nonce":35127}]},"public_addr":"172.21.2.136:6801/35127","cluster_addr":"172.21.2.136:6803/35127","heartbeat_back_addr":"172.21.2.136:6807/35127","heartbeat_front_addr":"172.21.2.136:6805/35127","state":["exists","up"]},{"osd":5,"uuid":"04e6b644-75ed-4a4b-bad0-c3c6ec60db04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6808","nonce":35128},{"type":"v1","addr":"172.21.2.136:6809","nonce":35128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6810","nonce":35128},{"type":"v1","addr":"172.21.2.136:6811","nonce":35128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6814","nonce":35128},{"type":"v1","addr":"172.21.2.136:6815","nonce":35128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6812","nonce":35128},{"type":"v1","addr":"172.21.2.136:6813","nonce":35128}]},"public_addr":"172.21.2.136:6809/35128","cluster_addr":"172.21.2.136:6811/35128","heartbeat_back_addr":"172.21.2.136:6815/35128","heartbeat_front_addr":"172.21.2.136:6813/35128","state":["exists","up"]},{"osd":6,"uuid":"1bef0f21-9f6a-4580-8618-34b3c99bdd16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6816","nonce":35129},{"type":"v1","addr":"172.21.2.136:6817","nonce":35129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6818","nonce":35129},{"type":"v1","addr":"172.21.2.136:6819","nonce":35129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6823","nonce":35129},{"type":"v1","addr":"172.21.2.136:6825","nonce":35129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6820","nonce":35129},{"type":"v1","addr":"172.21.2.136:6821","nonce":35129}]},"public_addr":"172.21.2.136:6817/35129","cluster_addr":"172.21.2.136:6819/35129","heartbeat_back_addr":"172.21.2.136:6825/35129","heartbeat_front_addr":"172.21.2.136:6821/35129","state":["exists","up"]},{"osd":7,"uuid":"938c8347-9c90-48f7-bef9-fdb6aad88bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":218,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6822","nonce":35130},{"type":"v1","addr":"172.21.2.136:6824","nonce":35130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6826","nonce":35130},{"type":"v1","addr":"172.21.2.136:6827","nonce":35130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6830","nonce":35130},{"type":"v1","addr":"172.21.2.136:6831","nonce":35130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.136:6828","nonce":35130},{"type":"v1","addr":"172.21.2.136:6829","nonce":35130}]},"public_addr":"172.21.2.136:6824/35130","cluster_addr":"172.21.2.136:6827/35130","heartbeat_back_addr":"172.21.2.136:6831/35130","heartbeat_front_addr":"172.21.2.136:6829/35130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.721854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:32.685787+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.339122+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:09.743044+0000","dead_epoch":18},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.661501+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.717093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:34.439797+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:42:33.618674+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"2.21","osds":[6,7]}],"pg_upmap_items":[{"pgid":"1.8","mappings":[{"from":5,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":5}]},{"pgid":"3.14","mappings":[{"from":4,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.1d","mappings":[{"from":4,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"isaprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"2","m":"1","name":"isaprofile","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:43:08.331 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats

Ignoring malparsed backtrace: 2021-03-11T17:43:16.547 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4
, instructed osd(s) 4 to deep-scrub
, 2021-03-11T17:43:16.771 INFO:tasks.ceph:Scrubbing osd.5

Ignoring malparsed backtrace: 2021-03-11T17:43:18.192 INFO:tasks.ceph:pgid 1.17 last_scrub_stamp 2021-03-11T17:42:44.288800+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.193 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2021-03-11T17:42:44.490732+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:18.194 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2021-03-11T17:43:06.994647+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=6, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:43:58.828 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2021-03-11T17:41:29.916342+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=41, tm_sec=29, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.829 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2021-03-11T17:42:22.136291+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:43:58.830 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-11T17:42:22.618589+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=42, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=43, tm_sec=13, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:44:20.744 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6
, 2021-03-11T17:44:20.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba036.front.sepia.ceph.com
, 2021-03-11T17:44:20.842 DEBUG:teuthology.orchestra.run.gibba036:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7

Ignoring malparsed backtrace: 2021-03-11T17:44:40.229 DEBUG:teuthology.orchestra.run.gibba020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'
, 2021-03-11T17:44:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2021-03-11T17:44:40.252 DEBUG:teuthology.orchestra.run.gibba036:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)'

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7efe9558a636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7efe9558a850]

Ignoring malparsed backtrace: 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:23.269099+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "op must wait for map",
, 2021-03-11T17:30:43.990 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:39.244808+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:            }
, 2021-03-11T17:30:44.019 INFO:teuthology.orchestra.run.gibba021.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.047 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 2.5578000000000001e-05
, 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.092 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:04.332370+0000",
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.8294000000000001e-05
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:36.266838+0000",
, 2021-03-11T17:30:44.162 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00018717899999999999

Ignoring malparsed backtrace: 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.220 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:18.298189+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.247 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:26.327424+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.308 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.8499999999999996e-06
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:19.278051+0000",
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.1860000000000002e-06
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                "events": [
, 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.666 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.793 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.6490000000000008e-06
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.848 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.281467+0000",
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.002115928
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.284813+0000",
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.6280000000000004e-06
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.901 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285183+0000",
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.358004558
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.647242+0000",
, 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.0051950349999999998
, 2021-03-11T17:30:44.928 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.982 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.657361+0000",
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.3720942899999999
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285400+0000",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:45.094 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "all_read",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.287193+0000",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.6780000000000003e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.288152+0000",
, 2021-03-11T17:30:45.165 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.19e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.318 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "started",
, 2021-03-11T17:30:45.324 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.658796+0000",
, 2021-03-11T17:30:45.356 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00040617099999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.672037+0000",
, 2021-03-11T17:30:45.433 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.383826947

Ignoring malparsed backtrace: 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client_addr": "172.21.2.109:0/1988433814",
, 2021-03-11T17:30:45.467 INFO:teuthology.orchestra.run.gibba021.stdout:                    "tid": 4305

Ignoring malparsed backtrace: 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312770+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.540 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.938 INFO:teuthology.orchestra.run.gibba021.stdout:            "description": "MOSDECSubOpRead(3.as5 66/65 ECSubRead(tid=171, to_read={3:597b892a:::gibba00935578-5:head=0,262144,0}, subchunks={3:597b892a:::gibba00935578-5:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "initiated_at": "2021-03-11T17:24:39.452422+0000",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "age": 366.48023686900001,

Ignoring malparsed backtrace: 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.8439999999999998e-06
, 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:45.984 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.026 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:26:59.886193+0000",
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.729e-06
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:00.746285+0000",
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.000117228
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:07.252214+0000",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 5.8248000000000002e-05

Ignoring malparsed backtrace: 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:50.856531+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.245 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.1360000000000002e-06
, 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.280 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:14.628365+0000",
, 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.58572948300000005
, 2021-03-11T17:30:46.307 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.338 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:25.345392+0000",
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.9695000000000003e-05
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:39.177008+0000",
, 2021-03-11T17:30:46.387 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 8.4209999999999995e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "complaint_time": 30,
, 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "num_blocked_ops": 0
, 2021-03-11T17:30:46.626 INFO:teuthology.orchestra.run.gibba042.stdout:}

Ignoring malparsed backtrace: 2021-03-11T17:30:46.788 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0
, 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.821 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.73910433499999995
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:40.188364+0000",
, 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.9284999999999998e-05
, 2021-03-11T17:30:46.913 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:39.199367+0000",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.000110982

Ignoring malparsed backtrace: 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:07.568680+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:42.633670+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.115 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:47.154 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.1200000000000003e-07
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.216 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.87492784899999998
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.598 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:47.599 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.504735+0000",
, 2021-03-11T17:30:47.600 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00034803500000000001

Ignoring malparsed backtrace: 2021-03-11T17:30:47.639 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.505896+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:28.309874+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.801 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "started",
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.891e-06
, 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.850 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:30:47.945 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.55334778600000001,
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.013 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 156.282001691,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.52771849299999996,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.040 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.913852+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.103 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.104 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.107 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.139 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.44398321800000001
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:48.503 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:12.226041+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.555 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.67811220299999997
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.911581+0000",
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 2.8609999999999998e-06
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.658 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.915416+0000",
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.96053252600000005
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.692 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 1.009925929,
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:31.252251+0000",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 5.5940000000000001e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 151.16067931399999,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.65716484600000002,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.783 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.981863+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.919 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.293235+0000",
, 2021-03-11T17:30:49.258 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:49.308 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.295228+0000",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 9.1880000000000008e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.297040+0000",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.195e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.273488+0000",
, 2021-03-11T17:30:49.393 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00052005300000000003

Ignoring malparsed backtrace: 2021-03-11T17:30:49.421 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.312417+0000",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00028353599999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.449 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:54.313040+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.640 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.97719924700000005
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:49.742 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:47.237408+0000",
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 4.8742000000000002e-05
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:30:03.359107+0000",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "all_read",
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:23.367815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.238 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                    }
, 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                ]
, 2021-03-11T17:30:50.269 INFO:teuthology.orchestra.run.gibba009.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:50.327 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 1.1769999999999999e-06
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.378 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:56.908602+0000",
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.000298277
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.387745+0000",
, 2021-03-11T17:30:50.410 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00056298700000000004

Ignoring malparsed backtrace: 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.504802+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.471 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 7.1303999999999994e-05
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.317917+0000",
, 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00097526100000000005
, 2021-03-11T17:30:50.518 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.553 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.713388+0000",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 5.6867999999999998e-05

Ignoring malparsed backtrace: 2021-03-11T17:31:14.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 1490353651736 got 1490353651736 for osd.0
, 2021-03-11T17:31:14.876 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-03-11T17:31:15.095 INFO:teuthology.orchestra.run.gibba009.stdout:73014444126

Ignoring malparsed backtrace: 2021-03-11T17:32:05.812 INFO:tasks.ceph:Scrubbing osd.2
, 2021-03-11T17:32:05.812 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:32:05.932 INFO:teuthology.orchestra.run.gibba009.stdout:{

Ignoring malparsed backtrace: 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2021-03-11T17:30:51.841574+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=51, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2021-03-11T17:30:56.099163+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=56, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.789 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-03-11T17:31:17.328179+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=31, tm_sec=17, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:32:51.247 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-11T17:32:51.322 INFO:tasks.ceph.osd.10:Stopped
, 2021-03-11T17:32:51.323 DEBUG:tasks.ceph.osd.11:waiting for process to exit

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/task/install/__init__.py", line 619, in task
,     yield
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks

Ignoring malparsed backtrace: 2021-03-11T17:33:59.604 DEBUG:teuthology.orchestra.run.gibba009:> test -e /home/ubuntu/cephtest/archive/coredump
, 2021-03-11T17:33:59.637 WARNING:teuthology.task.internal:Found coredumps on ubuntu@gibba009.front.sepia.ceph.com, flagging run as failed
, 2021-03-11T17:33:59.639 DEBUG:teuthology.orchestra.run.gibba021:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7efe9558a636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7efe9558a850]

Ignoring malparsed backtrace: 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:23.269099+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "op must wait for map",
, 2021-03-11T17:30:43.990 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:39.244808+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:            }
, 2021-03-11T17:30:44.019 INFO:teuthology.orchestra.run.gibba021.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.047 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 2.5578000000000001e-05
, 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.092 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:04.332370+0000",
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.8294000000000001e-05
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:36.266838+0000",
, 2021-03-11T17:30:44.162 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00018717899999999999

Ignoring malparsed backtrace: 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.220 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:18.298189+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.247 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:26.327424+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.308 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.8499999999999996e-06
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:19.278051+0000",
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.1860000000000002e-06
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                "events": [
, 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.666 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.793 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.6490000000000008e-06
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.848 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.281467+0000",
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.002115928
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.284813+0000",
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.6280000000000004e-06
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.901 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285183+0000",
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.358004558
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.647242+0000",
, 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.0051950349999999998
, 2021-03-11T17:30:44.928 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.982 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.657361+0000",
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.3720942899999999
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285400+0000",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:45.094 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "all_read",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.287193+0000",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.6780000000000003e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.288152+0000",
, 2021-03-11T17:30:45.165 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.19e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.318 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "started",
, 2021-03-11T17:30:45.324 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.658796+0000",
, 2021-03-11T17:30:45.356 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00040617099999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.672037+0000",
, 2021-03-11T17:30:45.433 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.383826947

Ignoring malparsed backtrace: 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client_addr": "172.21.2.109:0/1988433814",
, 2021-03-11T17:30:45.467 INFO:teuthology.orchestra.run.gibba021.stdout:                    "tid": 4305

Ignoring malparsed backtrace: 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312770+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.540 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.938 INFO:teuthology.orchestra.run.gibba021.stdout:            "description": "MOSDECSubOpRead(3.as5 66/65 ECSubRead(tid=171, to_read={3:597b892a:::gibba00935578-5:head=0,262144,0}, subchunks={3:597b892a:::gibba00935578-5:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "initiated_at": "2021-03-11T17:24:39.452422+0000",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "age": 366.48023686900001,

Ignoring malparsed backtrace: 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.8439999999999998e-06
, 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:45.984 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.026 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:26:59.886193+0000",
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.729e-06
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:00.746285+0000",
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.000117228
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:07.252214+0000",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 5.8248000000000002e-05

Ignoring malparsed backtrace: 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:50.856531+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.245 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.1360000000000002e-06
, 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.280 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:14.628365+0000",
, 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.58572948300000005
, 2021-03-11T17:30:46.307 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.338 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:25.345392+0000",
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.9695000000000003e-05
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:39.177008+0000",
, 2021-03-11T17:30:46.387 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 8.4209999999999995e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "complaint_time": 30,
, 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "num_blocked_ops": 0
, 2021-03-11T17:30:46.626 INFO:teuthology.orchestra.run.gibba042.stdout:}

Ignoring malparsed backtrace: 2021-03-11T17:30:46.788 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0
, 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.821 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.73910433499999995
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:40.188364+0000",
, 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.9284999999999998e-05
, 2021-03-11T17:30:46.913 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:39.199367+0000",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.000110982

Ignoring malparsed backtrace: 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:07.568680+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:42.633670+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.115 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:47.154 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.1200000000000003e-07
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.216 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.87492784899999998
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.598 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:47.599 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.504735+0000",
, 2021-03-11T17:30:47.600 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00034803500000000001

Ignoring malparsed backtrace: 2021-03-11T17:30:47.639 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.505896+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:28.309874+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.801 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "started",
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.891e-06
, 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.850 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:30:47.945 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.55334778600000001,
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.013 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 156.282001691,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.52771849299999996,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.040 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.913852+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.103 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.104 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.107 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.139 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.44398321800000001
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:48.503 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:12.226041+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.555 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.67811220299999997
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.911581+0000",
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 2.8609999999999998e-06
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.658 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.915416+0000",
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.96053252600000005
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.692 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 1.009925929,
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:31.252251+0000",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 5.5940000000000001e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 151.16067931399999,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.65716484600000002,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.783 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.981863+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.919 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.293235+0000",
, 2021-03-11T17:30:49.258 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:49.308 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.295228+0000",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 9.1880000000000008e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.297040+0000",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.195e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.273488+0000",
, 2021-03-11T17:30:49.393 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00052005300000000003

Ignoring malparsed backtrace: 2021-03-11T17:30:49.421 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.312417+0000",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00028353599999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.449 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:54.313040+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.640 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.97719924700000005
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:49.742 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:47.237408+0000",
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 4.8742000000000002e-05
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:30:03.359107+0000",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "all_read",
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:23.367815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.238 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                    }
, 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                ]
, 2021-03-11T17:30:50.269 INFO:teuthology.orchestra.run.gibba009.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:50.327 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 1.1769999999999999e-06
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.378 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:56.908602+0000",
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.000298277
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.387745+0000",
, 2021-03-11T17:30:50.410 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00056298700000000004

Ignoring malparsed backtrace: 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.504802+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.471 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 7.1303999999999994e-05
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.317917+0000",
, 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00097526100000000005
, 2021-03-11T17:30:50.518 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.553 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.713388+0000",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 5.6867999999999998e-05

Ignoring malparsed backtrace: 2021-03-11T17:31:14.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 1490353651736 got 1490353651736 for osd.0
, 2021-03-11T17:31:14.876 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-03-11T17:31:15.095 INFO:teuthology.orchestra.run.gibba009.stdout:73014444126

Ignoring malparsed backtrace: 2021-03-11T17:32:05.812 INFO:tasks.ceph:Scrubbing osd.2
, 2021-03-11T17:32:05.812 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:32:05.932 INFO:teuthology.orchestra.run.gibba009.stdout:{

Ignoring malparsed backtrace: 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2021-03-11T17:30:51.841574+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=51, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2021-03-11T17:30:56.099163+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=56, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.789 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-03-11T17:31:17.328179+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=31, tm_sec=17, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:32:51.247 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-11T17:32:51.322 INFO:tasks.ceph.osd.10:Stopped
, 2021-03-11T17:32:51.323 DEBUG:tasks.ceph.osd.11:waiting for process to exit

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/task/install/__init__.py", line 619, in task
,     yield
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks

Ignoring malparsed backtrace: 2021-03-11T17:33:59.604 DEBUG:teuthology.orchestra.run.gibba009:> test -e /home/ubuntu/cephtest/archive/coredump
, 2021-03-11T17:33:59.637 WARNING:teuthology.task.internal:Found coredumps on ubuntu@gibba009.front.sepia.ceph.com, flagging run as failed
, 2021-03-11T17:33:59.639 DEBUG:teuthology.orchestra.run.gibba021:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7efe9558a636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7efe9558a850]

Ignoring malparsed backtrace: 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:23.269099+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "op must wait for map",
, 2021-03-11T17:30:43.990 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:39.244808+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:            }
, 2021-03-11T17:30:44.019 INFO:teuthology.orchestra.run.gibba021.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.047 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 2.5578000000000001e-05
, 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.092 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:04.332370+0000",
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.8294000000000001e-05
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:36.266838+0000",
, 2021-03-11T17:30:44.162 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00018717899999999999

Ignoring malparsed backtrace: 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.220 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:18.298189+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.247 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:26.327424+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.308 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.8499999999999996e-06
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:19.278051+0000",
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.1860000000000002e-06
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                "events": [
, 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.666 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.793 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.6490000000000008e-06
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.848 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.281467+0000",
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.002115928
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.284813+0000",
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.6280000000000004e-06
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.901 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285183+0000",
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.358004558
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.647242+0000",
, 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.0051950349999999998
, 2021-03-11T17:30:44.928 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.982 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.657361+0000",
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.3720942899999999
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285400+0000",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:45.094 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "all_read",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.287193+0000",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.6780000000000003e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.288152+0000",
, 2021-03-11T17:30:45.165 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.19e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.318 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "started",
, 2021-03-11T17:30:45.324 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.658796+0000",
, 2021-03-11T17:30:45.356 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00040617099999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.672037+0000",
, 2021-03-11T17:30:45.433 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.383826947

Ignoring malparsed backtrace: 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client_addr": "172.21.2.109:0/1988433814",
, 2021-03-11T17:30:45.467 INFO:teuthology.orchestra.run.gibba021.stdout:                    "tid": 4305

Ignoring malparsed backtrace: 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312770+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.540 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.938 INFO:teuthology.orchestra.run.gibba021.stdout:            "description": "MOSDECSubOpRead(3.as5 66/65 ECSubRead(tid=171, to_read={3:597b892a:::gibba00935578-5:head=0,262144,0}, subchunks={3:597b892a:::gibba00935578-5:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "initiated_at": "2021-03-11T17:24:39.452422+0000",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "age": 366.48023686900001,

Ignoring malparsed backtrace: 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.8439999999999998e-06
, 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:45.984 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.026 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:26:59.886193+0000",
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.729e-06
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:00.746285+0000",
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.000117228
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:07.252214+0000",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 5.8248000000000002e-05

Ignoring malparsed backtrace: 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:50.856531+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.245 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.1360000000000002e-06
, 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.280 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:14.628365+0000",
, 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.58572948300000005
, 2021-03-11T17:30:46.307 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.338 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:25.345392+0000",
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.9695000000000003e-05
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:39.177008+0000",
, 2021-03-11T17:30:46.387 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 8.4209999999999995e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "complaint_time": 30,
, 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "num_blocked_ops": 0
, 2021-03-11T17:30:46.626 INFO:teuthology.orchestra.run.gibba042.stdout:}

Ignoring malparsed backtrace: 2021-03-11T17:30:46.788 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0
, 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.821 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.73910433499999995
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:40.188364+0000",
, 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.9284999999999998e-05
, 2021-03-11T17:30:46.913 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:39.199367+0000",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.000110982

Ignoring malparsed backtrace: 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:07.568680+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:42.633670+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.115 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:47.154 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.1200000000000003e-07
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.216 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.87492784899999998
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.598 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:47.599 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.504735+0000",
, 2021-03-11T17:30:47.600 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00034803500000000001

Ignoring malparsed backtrace: 2021-03-11T17:30:47.639 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.505896+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:28.309874+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.801 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "started",
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.891e-06
, 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.850 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:30:47.945 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.55334778600000001,
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.013 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 156.282001691,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.52771849299999996,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.040 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.913852+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.103 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.104 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.107 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.139 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.44398321800000001
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:48.503 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:12.226041+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.555 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.67811220299999997
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.911581+0000",
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 2.8609999999999998e-06
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.658 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.915416+0000",
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.96053252600000005
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.692 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 1.009925929,
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:31.252251+0000",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 5.5940000000000001e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 151.16067931399999,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.65716484600000002,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.783 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.981863+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.919 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.293235+0000",
, 2021-03-11T17:30:49.258 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:49.308 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.295228+0000",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 9.1880000000000008e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.297040+0000",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.195e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.273488+0000",
, 2021-03-11T17:30:49.393 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00052005300000000003

Ignoring malparsed backtrace: 2021-03-11T17:30:49.421 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.312417+0000",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00028353599999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.449 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:54.313040+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.640 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.97719924700000005
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:49.742 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:47.237408+0000",
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 4.8742000000000002e-05
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:30:03.359107+0000",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "all_read",
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:23.367815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.238 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                    }
, 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                ]
, 2021-03-11T17:30:50.269 INFO:teuthology.orchestra.run.gibba009.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:50.327 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 1.1769999999999999e-06
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.378 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:56.908602+0000",
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.000298277
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.387745+0000",
, 2021-03-11T17:30:50.410 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00056298700000000004

Ignoring malparsed backtrace: 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.504802+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.471 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 7.1303999999999994e-05
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.317917+0000",
, 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00097526100000000005
, 2021-03-11T17:30:50.518 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.553 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.713388+0000",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 5.6867999999999998e-05

Ignoring malparsed backtrace: 2021-03-11T17:31:14.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 1490353651736 got 1490353651736 for osd.0
, 2021-03-11T17:31:14.876 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-03-11T17:31:15.095 INFO:teuthology.orchestra.run.gibba009.stdout:73014444126

Ignoring malparsed backtrace: 2021-03-11T17:32:05.812 INFO:tasks.ceph:Scrubbing osd.2
, 2021-03-11T17:32:05.812 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:32:05.932 INFO:teuthology.orchestra.run.gibba009.stdout:{

Ignoring malparsed backtrace: 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2021-03-11T17:30:51.841574+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=51, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2021-03-11T17:30:56.099163+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=56, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.789 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-03-11T17:31:17.328179+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=31, tm_sec=17, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:32:51.247 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-11T17:32:51.322 INFO:tasks.ceph.osd.10:Stopped
, 2021-03-11T17:32:51.323 DEBUG:tasks.ceph.osd.11:waiting for process to exit

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/task/install/__init__.py", line 619, in task
,     yield
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks

Ignoring malparsed backtrace: 2021-03-11T17:33:59.604 DEBUG:teuthology.orchestra.run.gibba009:> test -e /home/ubuntu/cephtest/archive/coredump
, 2021-03-11T17:33:59.637 WARNING:teuthology.task.internal:Found coredumps on ubuntu@gibba009.front.sepia.ceph.com, flagging run as failed
, 2021-03-11T17:33:59.639 DEBUG:teuthology.orchestra.run.gibba021:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7efe9558a636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7efe9558a850]

Ignoring malparsed backtrace: 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:23.269099+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "op must wait for map",
, 2021-03-11T17:30:43.990 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:39.244808+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:            }
, 2021-03-11T17:30:44.019 INFO:teuthology.orchestra.run.gibba021.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.047 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 2.5578000000000001e-05
, 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.092 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:04.332370+0000",
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.8294000000000001e-05
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:36.266838+0000",
, 2021-03-11T17:30:44.162 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00018717899999999999

Ignoring malparsed backtrace: 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.220 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:18.298189+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.247 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:26.327424+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.308 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.8499999999999996e-06
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:19.278051+0000",
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.1860000000000002e-06
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                "events": [
, 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.666 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.793 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.6490000000000008e-06
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.848 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.281467+0000",
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.002115928
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.284813+0000",
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.6280000000000004e-06
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.901 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285183+0000",
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.358004558
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.647242+0000",
, 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.0051950349999999998
, 2021-03-11T17:30:44.928 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.982 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.657361+0000",
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.3720942899999999
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285400+0000",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:45.094 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "all_read",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.287193+0000",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.6780000000000003e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.288152+0000",
, 2021-03-11T17:30:45.165 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.19e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.318 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "started",
, 2021-03-11T17:30:45.324 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.658796+0000",
, 2021-03-11T17:30:45.356 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00040617099999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.672037+0000",
, 2021-03-11T17:30:45.433 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.383826947

Ignoring malparsed backtrace: 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client_addr": "172.21.2.109:0/1988433814",
, 2021-03-11T17:30:45.467 INFO:teuthology.orchestra.run.gibba021.stdout:                    "tid": 4305

Ignoring malparsed backtrace: 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312770+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.540 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.938 INFO:teuthology.orchestra.run.gibba021.stdout:            "description": "MOSDECSubOpRead(3.as5 66/65 ECSubRead(tid=171, to_read={3:597b892a:::gibba00935578-5:head=0,262144,0}, subchunks={3:597b892a:::gibba00935578-5:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "initiated_at": "2021-03-11T17:24:39.452422+0000",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "age": 366.48023686900001,

Ignoring malparsed backtrace: 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.8439999999999998e-06
, 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:45.984 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.026 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:26:59.886193+0000",
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.729e-06
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:00.746285+0000",
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.000117228
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:07.252214+0000",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 5.8248000000000002e-05

Ignoring malparsed backtrace: 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:50.856531+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.245 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.1360000000000002e-06
, 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.280 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:14.628365+0000",
, 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.58572948300000005
, 2021-03-11T17:30:46.307 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.338 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:25.345392+0000",
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.9695000000000003e-05
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:39.177008+0000",
, 2021-03-11T17:30:46.387 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 8.4209999999999995e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "complaint_time": 30,
, 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "num_blocked_ops": 0
, 2021-03-11T17:30:46.626 INFO:teuthology.orchestra.run.gibba042.stdout:}

Ignoring malparsed backtrace: 2021-03-11T17:30:46.788 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0
, 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.821 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.73910433499999995
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:40.188364+0000",
, 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.9284999999999998e-05
, 2021-03-11T17:30:46.913 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:39.199367+0000",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.000110982

Ignoring malparsed backtrace: 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:07.568680+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:42.633670+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.115 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:47.154 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.1200000000000003e-07
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.216 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.87492784899999998
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.598 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:47.599 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.504735+0000",
, 2021-03-11T17:30:47.600 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00034803500000000001

Ignoring malparsed backtrace: 2021-03-11T17:30:47.639 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.505896+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:28.309874+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.801 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "started",
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.891e-06
, 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.850 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:30:47.945 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.55334778600000001,
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.013 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 156.282001691,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.52771849299999996,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.040 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.913852+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.103 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.104 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.107 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.139 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.44398321800000001
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:48.503 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:12.226041+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.555 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.67811220299999997
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.911581+0000",
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 2.8609999999999998e-06
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.658 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.915416+0000",
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.96053252600000005
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.692 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 1.009925929,
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:31.252251+0000",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 5.5940000000000001e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 151.16067931399999,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.65716484600000002,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.783 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.981863+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.919 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.293235+0000",
, 2021-03-11T17:30:49.258 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:49.308 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.295228+0000",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 9.1880000000000008e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.297040+0000",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.195e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.273488+0000",
, 2021-03-11T17:30:49.393 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00052005300000000003

Ignoring malparsed backtrace: 2021-03-11T17:30:49.421 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.312417+0000",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00028353599999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.449 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:54.313040+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.640 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.97719924700000005
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:49.742 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:47.237408+0000",
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 4.8742000000000002e-05
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:30:03.359107+0000",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "all_read",
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:23.367815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.238 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                    }
, 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                ]
, 2021-03-11T17:30:50.269 INFO:teuthology.orchestra.run.gibba009.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:50.327 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 1.1769999999999999e-06
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.378 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:56.908602+0000",
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.000298277
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.387745+0000",
, 2021-03-11T17:30:50.410 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00056298700000000004

Ignoring malparsed backtrace: 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.504802+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.471 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 7.1303999999999994e-05
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.317917+0000",
, 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00097526100000000005
, 2021-03-11T17:30:50.518 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.553 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.713388+0000",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 5.6867999999999998e-05

Ignoring malparsed backtrace: 2021-03-11T17:31:14.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 1490353651736 got 1490353651736 for osd.0
, 2021-03-11T17:31:14.876 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-03-11T17:31:15.095 INFO:teuthology.orchestra.run.gibba009.stdout:73014444126

Ignoring malparsed backtrace: 2021-03-11T17:32:05.812 INFO:tasks.ceph:Scrubbing osd.2
, 2021-03-11T17:32:05.812 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:32:05.932 INFO:teuthology.orchestra.run.gibba009.stdout:{

Ignoring malparsed backtrace: 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2021-03-11T17:30:51.841574+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=51, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2021-03-11T17:30:56.099163+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=56, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.789 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-03-11T17:31:17.328179+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=31, tm_sec=17, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:32:51.247 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-11T17:32:51.322 INFO:tasks.ceph.osd.10:Stopped
, 2021-03-11T17:32:51.323 DEBUG:tasks.ceph.osd.11:waiting for process to exit

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/task/install/__init__.py", line 619, in task
,     yield
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks

Ignoring malparsed backtrace: 2021-03-11T17:33:59.604 DEBUG:teuthology.orchestra.run.gibba009:> test -e /home/ubuntu/cephtest/archive/coredump
, 2021-03-11T17:33:59.637 WARNING:teuthology.task.internal:Found coredumps on ubuntu@gibba009.front.sepia.ceph.com, flagging run as failed
, 2021-03-11T17:33:59.639 DEBUG:teuthology.orchestra.run.gibba021:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7efe9558a636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7efe9558a850]

Ignoring malparsed backtrace: 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:23.269099+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "op must wait for map",
, 2021-03-11T17:30:43.990 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:39.244808+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:            }
, 2021-03-11T17:30:44.019 INFO:teuthology.orchestra.run.gibba021.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.047 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 2.5578000000000001e-05
, 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.092 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:04.332370+0000",
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.8294000000000001e-05
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:36.266838+0000",
, 2021-03-11T17:30:44.162 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00018717899999999999

Ignoring malparsed backtrace: 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.220 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:18.298189+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.247 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:26.327424+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.308 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.8499999999999996e-06
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:19.278051+0000",
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.1860000000000002e-06
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                "events": [
, 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.666 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.793 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.6490000000000008e-06
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.848 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.281467+0000",
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.002115928
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.284813+0000",
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.6280000000000004e-06
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.901 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285183+0000",
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.358004558
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.647242+0000",
, 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.0051950349999999998
, 2021-03-11T17:30:44.928 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.982 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.657361+0000",
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.3720942899999999
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285400+0000",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:45.094 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "all_read",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.287193+0000",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.6780000000000003e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.288152+0000",
, 2021-03-11T17:30:45.165 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.19e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.318 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "started",
, 2021-03-11T17:30:45.324 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.658796+0000",
, 2021-03-11T17:30:45.356 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00040617099999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.672037+0000",
, 2021-03-11T17:30:45.433 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.383826947

Ignoring malparsed backtrace: 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client_addr": "172.21.2.109:0/1988433814",
, 2021-03-11T17:30:45.467 INFO:teuthology.orchestra.run.gibba021.stdout:                    "tid": 4305

Ignoring malparsed backtrace: 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312770+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.540 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.938 INFO:teuthology.orchestra.run.gibba021.stdout:            "description": "MOSDECSubOpRead(3.as5 66/65 ECSubRead(tid=171, to_read={3:597b892a:::gibba00935578-5:head=0,262144,0}, subchunks={3:597b892a:::gibba00935578-5:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "initiated_at": "2021-03-11T17:24:39.452422+0000",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "age": 366.48023686900001,

Ignoring malparsed backtrace: 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.8439999999999998e-06
, 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:45.984 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.026 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:26:59.886193+0000",
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.729e-06
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:00.746285+0000",
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.000117228
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:07.252214+0000",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 5.8248000000000002e-05

Ignoring malparsed backtrace: 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:50.856531+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.245 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.1360000000000002e-06
, 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.280 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:14.628365+0000",
, 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.58572948300000005
, 2021-03-11T17:30:46.307 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.338 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:25.345392+0000",
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.9695000000000003e-05
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:39.177008+0000",
, 2021-03-11T17:30:46.387 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 8.4209999999999995e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "complaint_time": 30,
, 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "num_blocked_ops": 0
, 2021-03-11T17:30:46.626 INFO:teuthology.orchestra.run.gibba042.stdout:}

Ignoring malparsed backtrace: 2021-03-11T17:30:46.788 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0
, 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.821 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.73910433499999995
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:40.188364+0000",
, 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.9284999999999998e-05
, 2021-03-11T17:30:46.913 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:39.199367+0000",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.000110982

Ignoring malparsed backtrace: 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:07.568680+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:42.633670+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.115 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:47.154 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.1200000000000003e-07
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.216 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.87492784899999998
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.598 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:47.599 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.504735+0000",
, 2021-03-11T17:30:47.600 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00034803500000000001

Ignoring malparsed backtrace: 2021-03-11T17:30:47.639 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.505896+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:28.309874+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.801 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "started",
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.891e-06
, 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.850 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:30:47.945 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.55334778600000001,
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.013 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 156.282001691,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.52771849299999996,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.040 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.913852+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.103 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.104 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.107 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.139 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.44398321800000001
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:48.503 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:12.226041+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.555 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.67811220299999997
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.911581+0000",
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 2.8609999999999998e-06
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.658 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.915416+0000",
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.96053252600000005
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.692 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 1.009925929,
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:31.252251+0000",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 5.5940000000000001e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 151.16067931399999,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.65716484600000002,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.783 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.981863+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.919 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.293235+0000",
, 2021-03-11T17:30:49.258 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:49.308 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.295228+0000",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 9.1880000000000008e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.297040+0000",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.195e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.273488+0000",
, 2021-03-11T17:30:49.393 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00052005300000000003

Ignoring malparsed backtrace: 2021-03-11T17:30:49.421 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.312417+0000",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00028353599999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.449 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:54.313040+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.640 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.97719924700000005
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:49.742 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:47.237408+0000",
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 4.8742000000000002e-05
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:30:03.359107+0000",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "all_read",
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:23.367815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.238 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                    }
, 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                ]
, 2021-03-11T17:30:50.269 INFO:teuthology.orchestra.run.gibba009.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:50.327 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 1.1769999999999999e-06
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.378 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:56.908602+0000",
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.000298277
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.387745+0000",
, 2021-03-11T17:30:50.410 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00056298700000000004

Ignoring malparsed backtrace: 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.504802+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.471 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 7.1303999999999994e-05
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.317917+0000",
, 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00097526100000000005
, 2021-03-11T17:30:50.518 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.553 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.713388+0000",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 5.6867999999999998e-05

Ignoring malparsed backtrace: 2021-03-11T17:31:14.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 1490353651736 got 1490353651736 for osd.0
, 2021-03-11T17:31:14.876 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-03-11T17:31:15.095 INFO:teuthology.orchestra.run.gibba009.stdout:73014444126

Ignoring malparsed backtrace: 2021-03-11T17:32:05.812 INFO:tasks.ceph:Scrubbing osd.2
, 2021-03-11T17:32:05.812 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:32:05.932 INFO:teuthology.orchestra.run.gibba009.stdout:{

Ignoring malparsed backtrace: 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2021-03-11T17:30:51.841574+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=51, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2021-03-11T17:30:56.099163+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=56, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.789 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-03-11T17:31:17.328179+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=31, tm_sec=17, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:32:51.247 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-11T17:32:51.322 INFO:tasks.ceph.osd.10:Stopped
, 2021-03-11T17:32:51.323 DEBUG:tasks.ceph.osd.11:waiting for process to exit

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/task/install/__init__.py", line 619, in task
,     yield
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks

Ignoring malparsed backtrace: 2021-03-11T17:33:59.604 DEBUG:teuthology.orchestra.run.gibba009:> test -e /home/ubuntu/cephtest/archive/coredump
, 2021-03-11T17:33:59.637 WARNING:teuthology.task.internal:Found coredumps on ubuntu@gibba009.front.sepia.ceph.com, flagging run as failed
, 2021-03-11T17:33:59.639 DEBUG:teuthology.orchestra.run.gibba021:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7efe9558a636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7efe9558a850]

Ignoring malparsed backtrace: 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:43.961 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:23.269099+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:43.989 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "op must wait for map",
, 2021-03-11T17:30:43.990 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:24:39.244808+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.018 INFO:teuthology.orchestra.run.gibba021.stdout:            }
, 2021-03-11T17:30:44.019 INFO:teuthology.orchestra.run.gibba021.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.046 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.047 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 2.5578000000000001e-05
, 2021-03-11T17:30:44.091 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.092 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:04.332370+0000",
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.8294000000000001e-05
, 2021-03-11T17:30:44.133 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.161 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:36.266838+0000",
, 2021-03-11T17:30:44.162 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00018717899999999999

Ignoring malparsed backtrace: 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.219 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:44.220 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:18.298189+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.247 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:44.248 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:26.327424+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.281 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.308 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.8499999999999996e-06
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.309 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:19.278051+0000",
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.1860000000000002e-06
, 2021-03-11T17:30:44.336 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                "events": [
, 2021-03-11T17:30:44.665 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:44.666 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                    }
, 2021-03-11T17:30:44.792 INFO:teuthology.orchestra.run.gibba021.stdout:                ]
, 2021-03-11T17:30:44.793 INFO:teuthology.orchestra.run.gibba021.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.6490000000000008e-06
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:44.822 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:44.848 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.281467+0000",
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.002115928
, 2021-03-11T17:30:44.849 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.284813+0000",
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.6280000000000004e-06
, 2021-03-11T17:30:44.876 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.901 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285183+0000",
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.358004558
, 2021-03-11T17:30:44.902 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.647242+0000",
, 2021-03-11T17:30:44.927 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.0051950349999999998
, 2021-03-11T17:30:44.928 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:44.982 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.657361+0000",
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.3720942899999999
, 2021-03-11T17:30:44.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.285400+0000",
, 2021-03-11T17:30:45.047 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:45.094 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "all_read",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.287193+0000",
, 2021-03-11T17:30:45.095 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.6780000000000003e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:45.164 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:41.288152+0000",
, 2021-03-11T17:30:45.165 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.19e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:45.318 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "started",
, 2021-03-11T17:30:45.324 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.658796+0000",
, 2021-03-11T17:30:45.356 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.00040617099999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",
, 2021-03-11T17:30:45.432 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:42.672037+0000",
, 2021-03-11T17:30:45.433 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 1.383826947

Ignoring malparsed backtrace: 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:45.466 INFO:teuthology.orchestra.run.gibba021.stdout:                    "client_addr": "172.21.2.109:0/1988433814",
, 2021-03-11T17:30:45.467 INFO:teuthology.orchestra.run.gibba021.stdout:                    "tid": 4305

Ignoring malparsed backtrace: 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "initiated",
, 2021-03-11T17:30:45.494 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312770+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.540 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "throttled",
, 2021-03-11T17:30:45.541 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:14.312815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:45.938 INFO:teuthology.orchestra.run.gibba021.stdout:            "description": "MOSDECSubOpRead(3.as5 66/65 ECSubRead(tid=171, to_read={3:597b892a:::gibba00935578-5:head=0,262144,0}, subchunks={3:597b892a:::gibba00935578-5:head=[0,1]}, attrs_to_read=))",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "initiated_at": "2021-03-11T17:24:39.452422+0000",
, 2021-03-11T17:30:45.939 INFO:teuthology.orchestra.run.gibba021.stdout:            "age": 366.48023686900001,

Ignoring malparsed backtrace: 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 7.8439999999999998e-06
, 2021-03-11T17:30:45.983 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:45.984 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.026 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:26:59.886193+0000",
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 4.729e-06
, 2021-03-11T17:30:46.027 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:00.746285+0000",
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.000117228
, 2021-03-11T17:30:46.059 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:07.252214+0000",
, 2021-03-11T17:30:46.089 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 5.8248000000000002e-05

Ignoring malparsed backtrace: 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",
, 2021-03-11T17:30:46.132 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:27:50.856531+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.195 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.244 INFO:teuthology.orchestra.run.gibba021.stdout:                    {
, 2021-03-11T17:30:46.245 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.1360000000000002e-06
, 2021-03-11T17:30:46.279 INFO:teuthology.orchestra.run.gibba021.stdout:                    },
, 2021-03-11T17:30:46.280 INFO:teuthology.orchestra.run.gibba021.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:14.628365+0000",
, 2021-03-11T17:30:46.306 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 0.58572948300000005
, 2021-03-11T17:30:46.307 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.338 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:29:25.345392+0000",
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 6.9695000000000003e-05
, 2021-03-11T17:30:46.339 INFO:teuthology.orchestra.run.gibba021.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:46.386 INFO:teuthology.orchestra.run.gibba021.stdout:                        "time": "2021-03-11T17:30:39.177008+0000",
, 2021-03-11T17:30:46.387 INFO:teuthology.orchestra.run.gibba021.stdout:                        "duration": 8.4209999999999995e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "complaint_time": 30,
, 2021-03-11T17:30:46.625 INFO:teuthology.orchestra.run.gibba042.stdout:    "num_blocked_ops": 0
, 2021-03-11T17:30:46.626 INFO:teuthology.orchestra.run.gibba042.stdout:}

Ignoring malparsed backtrace: 2021-03-11T17:30:46.788 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.789 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0
, 2021-03-11T17:30:46.820 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:46.821 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.73910433499999995
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:46.878 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:40.188364+0000",
, 2021-03-11T17:30:46.912 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.9284999999999998e-05
, 2021-03-11T17:30:46.913 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:39.199367+0000",
, 2021-03-11T17:30:46.952 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.000110982

Ignoring malparsed backtrace: 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",
, 2021-03-11T17:30:46.986 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:07.568680+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:47.016 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:42.633670+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.047 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.080 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.114 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.115 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:47.153 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:47.154 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 6.1200000000000003e-07
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.188 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.216 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.87492784899999998
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.217 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.598 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:47.599 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.504735+0000",
, 2021-03-11T17:30:47.600 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00034803500000000001

Ignoring malparsed backtrace: 2021-03-11T17:30:47.639 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:47.640 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:03.505896+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:47.679 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:28.309874+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.728 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:47.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.801 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "started",
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:47.802 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",

Ignoring malparsed backtrace: 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.891e-06
, 2021-03-11T17:30:47.849 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:47.850 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",
, 2021-03-11T17:30:47.894 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:30:47.945 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.55334778600000001,
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:47.946 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.013 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 156.282001691,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.52771849299999996,
, 2021-03-11T17:30:48.014 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.040 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.041 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.913852+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.103 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.104 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.107 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.139 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.44398321800000001
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.140 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.502 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:30:48.503 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:12.226041+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                "events": [
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.527 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.555 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.556 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.67811220299999997
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.587 INFO:teuthology.orchestra.run.gibba042.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.911581+0000",
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 2.8609999999999998e-06
, 2021-03-11T17:30:48.630 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.658 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:26:59.915416+0000",
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.96053252600000005
, 2021-03-11T17:30:48.659 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:48.692 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 1.009925929,
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {
, 2021-03-11T17:30:48.693 INFO:teuthology.orchestra.run.gibba042.stdout:                "flag_point": "reached pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:27:31.252251+0000",
, 2021-03-11T17:30:48.729 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 5.5940000000000001e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "age": 151.16067931399999,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "duration": 0.65716484600000002,
, 2021-03-11T17:30:48.754 INFO:teuthology.orchestra.run.gibba042.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.782 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "done",
, 2021-03-11T17:30:48.783 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:28:32.981863+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.819 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.842 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:48.884 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:48.918 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:48.919 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.257 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.293235+0000",
, 2021-03-11T17:30:49.258 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:49.308 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "all_read",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.295228+0000",
, 2021-03-11T17:30:49.320 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 9.1880000000000008e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:38.297040+0000",
, 2021-03-11T17:30:49.352 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 1.195e-06

Ignoring malparsed backtrace: 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "started",
, 2021-03-11T17:30:49.392 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.273488+0000",
, 2021-03-11T17:30:49.393 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00052005300000000003

Ignoring malparsed backtrace: 2021-03-11T17:30:49.421 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:39.312417+0000",
, 2021-03-11T17:30:49.422 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.00028353599999999998

Ignoring malparsed backtrace: 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.448 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.449 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:24:54.313040+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                "client_info": {
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client": "client.4434",
, 2021-03-11T17:30:49.473 INFO:teuthology.orchestra.run.gibba042.stdout:                    "client_addr": "172.21.2.109:0/1988433814",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.499 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.524 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.561 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    },
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                    {
, 2021-03-11T17:30:49.611 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "commit_sent",

Ignoring malparsed backtrace: 2021-03-11T17:30:49.640 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:                ]
, 2021-03-11T17:30:49.641 INFO:teuthology.orchestra.run.gibba042.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0.97719924700000005
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                    }
, 2021-03-11T17:30:49.688 INFO:teuthology.orchestra.run.gibba042.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:30:49.742 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:29:47.237408+0000",
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 4.8742000000000002e-05
, 2021-03-11T17:30:49.743 INFO:teuthology.orchestra.run.gibba042.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "event": "initiated",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "time": "2021-03-11T17:30:03.359107+0000",
, 2021-03-11T17:30:49.796 INFO:teuthology.orchestra.run.gibba042.stdout:                        "duration": 0

Ignoring malparsed backtrace: 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "all_read",
, 2021-03-11T17:30:50.192 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:23.367815+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.237 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.238 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                    }
, 2021-03-11T17:30:50.268 INFO:teuthology.orchestra.run.gibba009.stdout:                ]
, 2021-03-11T17:30:50.269 INFO:teuthology.orchestra.run.gibba009.stdout:            }

Ignoring malparsed backtrace: 2021-03-11T17:30:50.327 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 1.1769999999999999e-06
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.328 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.378 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:56.908602+0000",
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.000298277
, 2021-03-11T17:30:50.379 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.409 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.387745+0000",
, 2021-03-11T17:30:50.410 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00056298700000000004

Ignoring malparsed backtrace: 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",
, 2021-03-11T17:30:50.438 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:29:57.504802+0000",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.470 INFO:teuthology.orchestra.run.gibba009.stdout:                    {
, 2021-03-11T17:30:50.471 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 7.1303999999999994e-05
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    },
, 2021-03-11T17:30:50.493 INFO:teuthology.orchestra.run.gibba009.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.317917+0000",
, 2021-03-11T17:30:50.517 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 0.00097526100000000005
, 2021-03-11T17:30:50.518 INFO:teuthology.orchestra.run.gibba009.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:30:50.553 INFO:teuthology.orchestra.run.gibba009.stdout:                        "event": "queued_for_pg",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "time": "2021-03-11T17:30:14.713388+0000",
, 2021-03-11T17:30:50.554 INFO:teuthology.orchestra.run.gibba009.stdout:                        "duration": 5.6867999999999998e-05

Ignoring malparsed backtrace: 2021-03-11T17:31:14.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 1490353651736 got 1490353651736 for osd.0
, 2021-03-11T17:31:14.876 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1
, 2021-03-11T17:31:15.095 INFO:teuthology.orchestra.run.gibba009.stdout:73014444126

Ignoring malparsed backtrace: 2021-03-11T17:32:05.812 INFO:tasks.ceph:Scrubbing osd.2
, 2021-03-11T17:32:05.812 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:32:05.932 INFO:teuthology.orchestra.run.gibba009.stdout:{

Ignoring malparsed backtrace: 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2021-03-11T17:30:51.841574+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=51, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.788 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2021-03-11T17:30:56.099163+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=30, tm_sec=56, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:32:09.789 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-03-11T17:31:17.328179+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=31, tm_sec=17, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=32, tm_sec=4, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:32:51.247 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-11T17:32:51.322 INFO:tasks.ceph.osd.10:Stopped
, 2021-03-11T17:32:51.323 DEBUG:tasks.ceph.osd.11:waiting for process to exit

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/task/install/__init__.py", line 619, in task
,     yield
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/run_tasks.py", line 176, in run_tasks

Ignoring malparsed backtrace: 2021-03-11T17:33:59.604 DEBUG:teuthology.orchestra.run.gibba009:> test -e /home/ubuntu/cephtest/archive/coredump
, 2021-03-11T17:33:59.637 WARNING:teuthology.task.internal:Found coredumps on ubuntu@gibba009.front.sepia.ceph.com, flagging run as failed
, 2021-03-11T17:33:59.639 DEBUG:teuthology.orchestra.run.gibba021:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f66d3dda636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f66d3dda850]

Ignoring malparsed backtrace: 2021-03-11T17:02:09.688 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.688 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",
, 2021-03-11T17:02:09.688 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:35.080424+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.716 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.717 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.718 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.741 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.9728000000000003e-05
, 2021-03-11T17:02:09.741 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.741 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:09.785 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:35.702201+0000",
, 2021-03-11T17:02:09.785 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.3297999999999999e-05
, 2021-03-11T17:02:09.785 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:09.808 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",
, 2021-03-11T17:02:09.808 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:35.903609+0000",
, 2021-03-11T17:02:09.808 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.7372000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:02:09.830 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.830 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",
, 2021-03-11T17:02:09.830 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:36.113531+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.854 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.854 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.854 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.877 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:09.877 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.877 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:09.900 INFO:teuthology.orchestra.run.gibba029.stdout:                    "tid": 7734
, 2021-03-11T17:02:09.900 INFO:teuthology.orchestra.run.gibba029.stdout:                },
, 2021-03-11T17:02:09.901 INFO:teuthology.orchestra.run.gibba029.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:02:09.922 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:44.291735+0000",
, 2021-03-11T17:02:09.922 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 2.1678000000000001e-05
, 2021-03-11T17:02:09.922 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:09.948 INFO:teuthology.orchestra.run.gibba029.stdout:        }
, 2021-03-11T17:02:09.948 INFO:teuthology.orchestra.run.gibba029.stdout:    ]
, 2021-03-11T17:02:09.948 INFO:teuthology.orchestra.run.gibba029.stdout:}

Ignoring malparsed backtrace: 2021-03-11T17:02:10.327 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:58:26.875758+0000",
, 2021-03-11T17:02:10.327 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:10.327 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:10.351 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:58:26.914304+0000",
, 2021-03-11T17:02:10.351 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.038483092000000003
, 2021-03-11T17:02:10.351 INFO:teuthology.orchestra.run.gibba029.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:02:10.375 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",
, 2021-03-11T17:02:10.376 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:10.908086+0000",
, 2021-03-11T17:02:10.376 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 2.7968999999999999e-05

Ignoring malparsed backtrace: 2021-03-11T17:02:10.399 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "started",
, 2021-03-11T17:02:10.399 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:10.914856+0000",
, 2021-03-11T17:02:10.399 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.024848762999999999

Ignoring malparsed backtrace: 2021-03-11T17:02:10.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:02:10.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:41.138198+0000",
, 2021-03-11T17:02:10.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.0059484389999999998

Ignoring malparsed backtrace: 2021-03-11T17:02:10.448 INFO:teuthology.orchestra.run.gibba029.stdout:            }
, 2021-03-11T17:02:10.448 INFO:teuthology.orchestra.run.gibba029.stdout:        },
, 2021-03-11T17:02:10.448 INFO:teuthology.orchestra.run.gibba029.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:02:10.470 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client": "client.4572",
, 2021-03-11T17:02:10.471 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client_addr": "172.21.2.116:0/2452094959",
, 2021-03-11T17:02:10.471 INFO:teuthology.orchestra.run.gibba029.stdout:                    "tid": 7571

Ignoring malparsed backtrace: 2021-03-11T17:02:10.493 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.493 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "initiated",
, 2021-03-11T17:02:10.494 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:32.179292+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.516 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.516 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "initiated",
, 2021-03-11T17:02:10.516 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:38.087316+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.538 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.539 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "waiting for missing object",
, 2021-03-11T17:02:10.539 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:38.088509+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.562 INFO:teuthology.orchestra.run.gibba029.stdout:                "client_info": {
, 2021-03-11T17:02:10.562 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client": "client.4572",
, 2021-03-11T17:02:10.562 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client_addr": "172.21.2.116:0/2452094959",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.584 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.584 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.584 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.606 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.606 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.607 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.628 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.629 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.629 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.651 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.651 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.651 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.673 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.673 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.673 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.930 INFO:tasks.thrashosds:joining thrashosds
, 2021-03-11T17:02:10.904+0000 7f94028cd700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5  (PID: 35128) UID: 0
, 2021-03-11T17:02:10.951 INFO:teuthology.orchestra.run.gibba029.stdout:{

Ignoring malparsed backtrace: 2021-03-11T17:02:11.127 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.127 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "initiated",
, 2021-03-11T17:02:11.127 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:58:35.943479+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.150 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.150 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",
, 2021-03-11T17:02:11.151 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:38.891562+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.175 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.175 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:02:11.175 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:38.891726+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.200 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.201 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "started",
, 2021-03-11T17:02:11.201 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:39.231978+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.222 INFO:teuthology.orchestra.run.gibba029.stdout:                ]
, 2021-03-11T17:02:11.223 INFO:teuthology.orchestra.run.gibba029.stdout:            }
, 2021-03-11T17:02:11.223 INFO:teuthology.orchestra.run.gibba029.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:02:11.245 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.245 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.245 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.268 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.268 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.268 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.291 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.291 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.291 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.312 INFO:teuthology.orchestra.run.gibba029.stdout:            "description": "osd_op(client.4572.0:8599 3.fs0 3:f1e85c40:::gibba01641143-94:head [write 1859448~425558] snapc ba=[ba,b8,b5,b3,b2,ac,a7,a3,9c,9b,93,8c] ondisk+write+known_if_redirected e241)",
, 2021-03-11T17:02:11.313 INFO:teuthology.orchestra.run.gibba029.stdout:            "initiated_at": "2021-03-11T17:01:38.027124+0000",
, 2021-03-11T17:02:11.313 INFO:teuthology.orchestra.run.gibba029.stdout:            "age": 33.077382381,

Ignoring malparsed backtrace: 2021-03-11T17:02:11.334 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:11.335 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.335 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.356 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 9.5000000000000005e-06
, 2021-03-11T17:02:11.357 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.357 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.378 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00012094700000000001
, 2021-03-11T17:02:11.378 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.378 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.401 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 8.3714000000000005e-05
, 2021-03-11T17:02:11.401 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.401 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00040583399999999999
, 2021-03-11T17:02:11.425 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.425 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.447 INFO:teuthology.orchestra.run.gibba029.stdout:        {
, 2021-03-11T17:02:11.447 INFO:teuthology.orchestra.run.gibba029.stdout:            "description": "osd_op(client.4572.0:9245 3.es0 3:71948077:::gibba01641143-693:head [copy-from ver 206] snapc d7=[d7,d5,d1,d0,ce,c7,c2,93] ondisk+write+known_if_redirected e270)",
, 2021-03-11T17:02:11.448 INFO:teuthology.orchestra.run.gibba029.stdout:            "initiated_at": "2021-03-11T17:02:06.907966+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.470 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 2.0030282439999998
, 2021-03-11T17:02:11.470 INFO:teuthology.orchestra.run.gibba029.stdout:                    }
, 2021-03-11T17:02:11.470 INFO:teuthology.orchestra.run.gibba029.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:02:11.592 INFO:teuthology.orchestra.run.gibba029.stdout:    "num_ops": 0
, 2021-03-11T17:02:11.592 INFO:teuthology.orchestra.run.gibba029.stdout:}
, 2021-03-11T17:02:11.598 DEBUG:teuthology.orchestra.run.gibba029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops

Ignoring malparsed backtrace: 2021-03-11T17:02:11.822 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.822 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.822 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.845 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.845 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.845 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.868 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.3759999999999999e-06
, 2021-03-11T17:02:11.868 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.868 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.890 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00075816700000000002
, 2021-03-11T17:02:11.890 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.891 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.912 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 4.2586000000000002e-05
, 2021-03-11T17:02:11.913 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.913 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.936 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00071467500000000001
, 2021-03-11T17:02:11.936 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.936 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.963 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:59.900788+0000",
, 2021-03-11T17:02:11.963 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:11.963 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:11.985 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:07.937528+0000",
, 2021-03-11T17:02:11.986 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 4.8640000000000004e-06
, 2021-03-11T17:02:11.986 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:12.009 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:07.940262+0000",
, 2021-03-11T17:02:12.010 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.0024267970000000001
, 2021-03-11T17:02:12.010 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:12.036 INFO:teuthology.orchestra.run.gibba029.stdout:            "duration": 0.0067861859999999996,
, 2021-03-11T17:02:12.036 INFO:teuthology.orchestra.run.gibba029.stdout:            "type_data": {
, 2021-03-11T17:02:12.036 INFO:teuthology.orchestra.run.gibba029.stdout:                "flag_point": "commit sent; apply or cleanup",

Ignoring malparsed backtrace: 2021-03-11T17:02:12.062 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "all_read",
, 2021-03-11T17:02:12.062 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:17.958750+0000",
, 2021-03-11T17:02:12.063 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 7.5850000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:02:12.088 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:02:12.088 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:17.966891+0000",
, 2021-03-11T17:02:12.089 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00090775700000000003

Ignoring malparsed backtrace: 2021-03-11T17:02:12.115 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client": "client.4572",
, 2021-03-11T17:02:12.116 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client_addr": "172.21.2.116:0/2452094959",
, 2021-03-11T17:02:12.116 INFO:teuthology.orchestra.run.gibba029.stdout:                    "tid": 8518

Ignoring malparsed backtrace: 2021-03-11T17:02:12.142 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:12.142 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",
, 2021-03-11T17:02:12.143 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:33.075251+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:12.168 INFO:teuthology.orchestra.run.gibba029.stdout:                ]
, 2021-03-11T17:02:12.168 INFO:teuthology.orchestra.run.gibba029.stdout:            }
, 2021-03-11T17:02:12.169 INFO:teuthology.orchestra.run.gibba029.stdout:        }

Ignoring malparsed backtrace: 2021-03-11T17:02:22.974+0000 7ffb03226700 -1 mon.a@0(leader).osd e278 definitely_dead 0
, marked in osd.6.
, 2021-03-11T17:02:23.078 INFO:tasks.thrashosds.thrasher:Added osd 6

Ignoring malparsed backtrace: 2021-03-11T17:02:41.793 INFO:teuthology.orchestra.run.gibba016.stdout:{"epoch":297,"fsid":"0288d25f-2798-4486-b9ed-6c1fa538c347","created":"2021-03-11T16:57:25.062737+0000","modified":"2021-03-11T17:02:41.330526+0000","last_up_change":"2021-03-11T17:02:19.040478+0000","last_in_change":"2021-03-11T17:02:23.058191+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T16:57:34.118119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"280","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"176","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T16:57:35.448919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":48,"pg_placement_num":48,"pg_placement_num_target":48,"pg_num_target":48,"pg_num_pending":48,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"142","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T16:57:46.854374+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"281","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"236","auid":0,"snap_mode":"selfmanaged","snap_seq":219,"snap_epoch":273,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ba2f09a4-7d62-494a-bc9a-b7dfc6d60bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":152,"up_from":187,"up_thru":295,"down_at":153,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6800","nonce":58253},{"type":"v1","addr":"172.21.2.116:6801","nonce":58253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6802","nonce":58253},{"type":"v1","addr":"172.21.2.116:6803","nonce":58253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6806","nonce":58253},{"type":"v1","addr":"172.21.2.116:6807","nonce":58253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6804","nonce":58253},{"type":"v1","addr":"172.21.2.116:6805","nonce":58253}]},"public_addr":"172.21.2.116:6801/58253","cluster_addr":"172.21.2.116:6803/58253","heartbeat_back_addr":"172.21.2.116:6807/58253","heartbeat_front_addr":"172.21.2.116:6805/58253","state":["exists","up"]},{"osd":1,"uuid":"577613bb-a808-4a5f-80d4-d71d0bb35676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":270,"up_from":277,"up_thru":291,"down_at":271,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6809","nonce":69535},{"type":"v1","addr":"172.21.2.116:6811","nonce":69535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6813","nonce":69535},{"type":"v1","addr":"172.21.2.116:6815","nonce":69535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6821","nonce":69535},{"type":"v1","addr":"172.21.2.116:6824","nonce":69535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6817","nonce":69535},{"type":"v1","addr":"172.21.2.116:6819","nonce":69535}]},"public_addr":"172.21.2.116:6811/69535","cluster_addr":"172.21.2.116:6815/69535","heartbeat_back_addr":"172.21.2.116:6824/69535","heartbeat_front_addr":"172.21.2.116:6819/69535","state":["exists","up"]},{"osd":2,"uuid":"c8fd4d3a-73f2-484d-8d9a-4bcc168398bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6808","nonce":35126},{"type":"v1","addr":"172.21.2.116:6810","nonce":35126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6812","nonce":35126},{"type":"v1","addr":"172.21.2.116:6814","nonce":35126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6820","nonce":35126},{"type":"v1","addr":"172.21.2.116:6823","nonce":35126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6816","nonce":35126},{"type":"v1","addr":"172.21.2.116:6818","nonce":35126}]},"public_addr":"172.21.2.116:6810/35126","cluster_addr":"172.21.2.116:6814/35126","heartbeat_back_addr":"172.21.2.116:6823/35126","heartbeat_front_addr":"172.21.2.116:6818/35126","state":["exists","up"]},{"osd":3,"uuid":"e0437b6a-c36d-4401-9cc0-fc64443fb0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":292,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6822","nonce":35127},{"type":"v1","addr":"172.21.2.116:6825","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6826","nonce":35127},{"type":"v1","addr":"172.21.2.116:6827","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6830","nonce":35127},{"type":"v1","addr":"172.21.2.116:6831","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6828","nonce":35127},{"type":"v1","addr":"172.21.2.116:6829","nonce":35127}]},"public_addr":"172.21.2.116:6825/35127","cluster_addr":"172.21.2.116:6827/35127","heartbeat_back_addr":"172.21.2.116:6831/35127","heartbeat_front_addr":"172.21.2.116:6829/35127","state":["exists","up"]},{"osd":4,"uuid":"adb93d6b-fdd1-4cf2-a670-0b579ff1556d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":293,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6800","nonce":35136},{"type":"v1","addr":"172.21.2.129:6801","nonce":35136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6802","nonce":35136},{"type":"v1","addr":"172.21.2.129:6803","nonce":35136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6806","nonce":35136},{"type":"v1","addr":"172.21.2.129:6807","nonce":35136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6804","nonce":35136},{"type":"v1","addr":"172.21.2.129:6805","nonce":35136}]},"public_addr":"172.21.2.129:6801/35136","cluster_addr":"172.21.2.129:6803/35136","heartbeat_back_addr":"172.21.2.129:6807/35136","heartbeat_front_addr":"172.21.2.129:6805/35136","state":["exists","up"]},{"osd":5,"uuid":"24747ad1-47a9-4a4a-b44b-8e95ac3dff7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":280,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6808","nonce":35137},{"type":"v1","addr":"172.21.2.129:6809","nonce":35137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6810","nonce":35137},{"type":"v1","addr":"172.21.2.129:6811","nonce":35137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6814","nonce":35137},{"type":"v1","addr":"172.21.2.129:6815","nonce":35137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6812","nonce":35137},{"type":"v1","addr":"172.21.2.129:6813","nonce":35137}]},"public_addr":"172.21.2.129:6809/35137","cluster_addr":"172.21.2.129:6811/35137","heartbeat_back_addr":"172.21.2.129:6815/35137","heartbeat_front_addr":"172.21.2.129:6813/35137","state":["exists","up"]},{"osd":6,"uuid":"d22cf1cd-03e3-4a34-9371-88c2fedca9a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":125,"up_from":136,"up_thru":294,"down_at":126,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6817","nonce":44841},{"type":"v1","addr":"172.21.2.129:6819","nonce":44841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6821","nonce":44841},{"type":"v1","addr":"172.21.2.129:6823","nonce":44841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6830","nonce":44841},{"type":"v1","addr":"172.21.2.129:6831","nonce":44841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6825","nonce":44841},{"type":"v1","addr":"172.21.2.129:6828","nonce":44841}]},"public_addr":"172.21.2.129:6819/44841","cluster_addr":"172.21.2.129:6823/44841","heartbeat_back_addr":"172.21.2.129:6831/44841","heartbeat_front_addr":"172.21.2.129:6828/44841","state":["exists","up"]},{"osd":7,"uuid":"acf539ae-1f07-403d-bb98-7b6ec25f2a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":288,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6816","nonce":35139},{"type":"v1","addr":"172.21.2.129:6818","nonce":35139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6820","nonce":35139},{"type":"v1","addr":"172.21.2.129:6822","nonce":35139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6827","nonce":35139},{"type":"v1","addr":"172.21.2.129:6829","nonce":35139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6824","nonce":35139},{"type":"v1","addr":"172.21.2.129:6826","nonce":35139}]},"public_addr":"172.21.2.129:6818/35139","cluster_addr":"172.21.2.129:6822/35139","heartbeat_back_addr":"172.21.2.129:6829/35139","heartbeat_front_addr":"172.21.2.129:6826/35139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:01:44.146927+0000","dead_epoch":153},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:33.859749+0000","dead_epoch":271},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.108809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.074493+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:33.962549+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.620024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.155841+0000","dead_epoch":126},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.574020+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":1,"to":2}]}],"pg_temp":[{"pgid":"3.c","osds":[5,3,2147483647]}],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":140,"length":1},{"begin":147,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:02:41.804 DEBUG:teuthology.orchestra.run.gibba016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats
, 2021-03-11T17:02:41.894+0000 7f8adceb1700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:02:48.019 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout
, 2021-03-11T17:02:48.020 DEBUG:teuthology.orchestra.run.gibba016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-03-11T17:02:48.110+0000 7f57b5c48700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:02:55.830 INFO:tasks.ceph:Scrubbing osd.6
, 2021-03-11T17:02:55.830 DEBUG:teuthology.orchestra.run.gibba016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:02:55.918+0000 7f1251d00700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:02:56.822 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2021-03-11T17:02:14.286275+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=14, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:02:56.823 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2021-03-11T17:02:48.781539+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=48, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:02:56.823 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2021-03-11T17:00:31.204784+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=0, tm_sec=31, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:03:37.397 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2021-03-11T17:01:44.368297+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=1, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:03:37.398 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2021-03-11T17:02:52.191793+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:03:37.399 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2021-03-11T17:01:22.260303+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=1, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:04:18.791 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-11T17:04:18.798 INFO:tasks.ceph.mon.b:Stopped
, 2021-03-11T17:04:18.799 INFO:tasks.ceph:Checking cluster log for badness...

Ignoring malparsed backtrace: 2021-03-11T17:05:14.992 INFO:teuthology.orchestra.run.gibba016.stdout:^* hv02.front.sepia.ceph.com     2   6   377    16   +613us[ +615us] +/-   30ms
, 2021-03-11T17:05:14.992 INFO:teuthology.orchestra.run.gibba016.stdout:^+ hv03.front.sepia.ceph.com     3   6   377    16  -1656us[-1656us] +/-   76ms
, 2021-03-11T17:05:14.992 INFO:teuthology.orchestra.run.gibba016.stdout:^+ hv04.front.sepia.ceph.com     3   6   377    15   +206us[ +206us] +/-   71ms

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f66d3dda636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f66d3dda850]

Ignoring malparsed backtrace: 2021-03-11T17:02:09.688 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.688 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",
, 2021-03-11T17:02:09.688 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:35.080424+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.716 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.717 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.718 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.741 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.9728000000000003e-05
, 2021-03-11T17:02:09.741 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.741 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:09.785 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:35.702201+0000",
, 2021-03-11T17:02:09.785 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.3297999999999999e-05
, 2021-03-11T17:02:09.785 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:09.808 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",
, 2021-03-11T17:02:09.808 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:35.903609+0000",
, 2021-03-11T17:02:09.808 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.7372000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:02:09.830 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.830 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",
, 2021-03-11T17:02:09.830 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:36.113531+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.854 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.854 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.854 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.877 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:09.877 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.877 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:09.900 INFO:teuthology.orchestra.run.gibba029.stdout:                    "tid": 7734
, 2021-03-11T17:02:09.900 INFO:teuthology.orchestra.run.gibba029.stdout:                },
, 2021-03-11T17:02:09.901 INFO:teuthology.orchestra.run.gibba029.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:02:09.922 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:44.291735+0000",
, 2021-03-11T17:02:09.922 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 2.1678000000000001e-05
, 2021-03-11T17:02:09.922 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:09.948 INFO:teuthology.orchestra.run.gibba029.stdout:        }
, 2021-03-11T17:02:09.948 INFO:teuthology.orchestra.run.gibba029.stdout:    ]
, 2021-03-11T17:02:09.948 INFO:teuthology.orchestra.run.gibba029.stdout:}

Ignoring malparsed backtrace: 2021-03-11T17:02:10.327 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:58:26.875758+0000",
, 2021-03-11T17:02:10.327 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:10.327 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:10.351 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:58:26.914304+0000",
, 2021-03-11T17:02:10.351 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.038483092000000003
, 2021-03-11T17:02:10.351 INFO:teuthology.orchestra.run.gibba029.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:02:10.375 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",
, 2021-03-11T17:02:10.376 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:10.908086+0000",
, 2021-03-11T17:02:10.376 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 2.7968999999999999e-05

Ignoring malparsed backtrace: 2021-03-11T17:02:10.399 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "started",
, 2021-03-11T17:02:10.399 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:10.914856+0000",
, 2021-03-11T17:02:10.399 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.024848762999999999

Ignoring malparsed backtrace: 2021-03-11T17:02:10.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:02:10.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:41.138198+0000",
, 2021-03-11T17:02:10.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.0059484389999999998

Ignoring malparsed backtrace: 2021-03-11T17:02:10.448 INFO:teuthology.orchestra.run.gibba029.stdout:            }
, 2021-03-11T17:02:10.448 INFO:teuthology.orchestra.run.gibba029.stdout:        },
, 2021-03-11T17:02:10.448 INFO:teuthology.orchestra.run.gibba029.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:02:10.470 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client": "client.4572",
, 2021-03-11T17:02:10.471 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client_addr": "172.21.2.116:0/2452094959",
, 2021-03-11T17:02:10.471 INFO:teuthology.orchestra.run.gibba029.stdout:                    "tid": 7571

Ignoring malparsed backtrace: 2021-03-11T17:02:10.493 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.493 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "initiated",
, 2021-03-11T17:02:10.494 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:32.179292+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.516 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.516 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "initiated",
, 2021-03-11T17:02:10.516 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:38.087316+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.538 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.539 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "waiting for missing object",
, 2021-03-11T17:02:10.539 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:38.088509+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.562 INFO:teuthology.orchestra.run.gibba029.stdout:                "client_info": {
, 2021-03-11T17:02:10.562 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client": "client.4572",
, 2021-03-11T17:02:10.562 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client_addr": "172.21.2.116:0/2452094959",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.584 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.584 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.584 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.606 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.606 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.607 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.628 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.629 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.629 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.651 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.651 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.651 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.673 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.673 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.673 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.930 INFO:tasks.thrashosds:joining thrashosds
, 2021-03-11T17:02:10.904+0000 7f94028cd700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5  (PID: 35128) UID: 0
, 2021-03-11T17:02:10.951 INFO:teuthology.orchestra.run.gibba029.stdout:{

Ignoring malparsed backtrace: 2021-03-11T17:02:11.127 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.127 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "initiated",
, 2021-03-11T17:02:11.127 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:58:35.943479+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.150 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.150 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",
, 2021-03-11T17:02:11.151 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:38.891562+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.175 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.175 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:02:11.175 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:38.891726+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.200 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.201 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "started",
, 2021-03-11T17:02:11.201 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:39.231978+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.222 INFO:teuthology.orchestra.run.gibba029.stdout:                ]
, 2021-03-11T17:02:11.223 INFO:teuthology.orchestra.run.gibba029.stdout:            }
, 2021-03-11T17:02:11.223 INFO:teuthology.orchestra.run.gibba029.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:02:11.245 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.245 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.245 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.268 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.268 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.268 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.291 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.291 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.291 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.312 INFO:teuthology.orchestra.run.gibba029.stdout:            "description": "osd_op(client.4572.0:8599 3.fs0 3:f1e85c40:::gibba01641143-94:head [write 1859448~425558] snapc ba=[ba,b8,b5,b3,b2,ac,a7,a3,9c,9b,93,8c] ondisk+write+known_if_redirected e241)",
, 2021-03-11T17:02:11.313 INFO:teuthology.orchestra.run.gibba029.stdout:            "initiated_at": "2021-03-11T17:01:38.027124+0000",
, 2021-03-11T17:02:11.313 INFO:teuthology.orchestra.run.gibba029.stdout:            "age": 33.077382381,

Ignoring malparsed backtrace: 2021-03-11T17:02:11.334 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:11.335 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.335 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.356 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 9.5000000000000005e-06
, 2021-03-11T17:02:11.357 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.357 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.378 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00012094700000000001
, 2021-03-11T17:02:11.378 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.378 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.401 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 8.3714000000000005e-05
, 2021-03-11T17:02:11.401 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.401 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00040583399999999999
, 2021-03-11T17:02:11.425 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.425 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.447 INFO:teuthology.orchestra.run.gibba029.stdout:        {
, 2021-03-11T17:02:11.447 INFO:teuthology.orchestra.run.gibba029.stdout:            "description": "osd_op(client.4572.0:9245 3.es0 3:71948077:::gibba01641143-693:head [copy-from ver 206] snapc d7=[d7,d5,d1,d0,ce,c7,c2,93] ondisk+write+known_if_redirected e270)",
, 2021-03-11T17:02:11.448 INFO:teuthology.orchestra.run.gibba029.stdout:            "initiated_at": "2021-03-11T17:02:06.907966+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.470 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 2.0030282439999998
, 2021-03-11T17:02:11.470 INFO:teuthology.orchestra.run.gibba029.stdout:                    }
, 2021-03-11T17:02:11.470 INFO:teuthology.orchestra.run.gibba029.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:02:11.592 INFO:teuthology.orchestra.run.gibba029.stdout:    "num_ops": 0
, 2021-03-11T17:02:11.592 INFO:teuthology.orchestra.run.gibba029.stdout:}
, 2021-03-11T17:02:11.598 DEBUG:teuthology.orchestra.run.gibba029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops

Ignoring malparsed backtrace: 2021-03-11T17:02:11.822 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.822 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.822 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.845 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.845 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.845 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.868 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.3759999999999999e-06
, 2021-03-11T17:02:11.868 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.868 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.890 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00075816700000000002
, 2021-03-11T17:02:11.890 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.891 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.912 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 4.2586000000000002e-05
, 2021-03-11T17:02:11.913 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.913 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.936 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00071467500000000001
, 2021-03-11T17:02:11.936 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.936 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.963 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:59.900788+0000",
, 2021-03-11T17:02:11.963 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:11.963 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:11.985 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:07.937528+0000",
, 2021-03-11T17:02:11.986 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 4.8640000000000004e-06
, 2021-03-11T17:02:11.986 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:12.009 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:07.940262+0000",
, 2021-03-11T17:02:12.010 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.0024267970000000001
, 2021-03-11T17:02:12.010 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:12.036 INFO:teuthology.orchestra.run.gibba029.stdout:            "duration": 0.0067861859999999996,
, 2021-03-11T17:02:12.036 INFO:teuthology.orchestra.run.gibba029.stdout:            "type_data": {
, 2021-03-11T17:02:12.036 INFO:teuthology.orchestra.run.gibba029.stdout:                "flag_point": "commit sent; apply or cleanup",

Ignoring malparsed backtrace: 2021-03-11T17:02:12.062 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "all_read",
, 2021-03-11T17:02:12.062 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:17.958750+0000",
, 2021-03-11T17:02:12.063 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 7.5850000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:02:12.088 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:02:12.088 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:17.966891+0000",
, 2021-03-11T17:02:12.089 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00090775700000000003

Ignoring malparsed backtrace: 2021-03-11T17:02:12.115 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client": "client.4572",
, 2021-03-11T17:02:12.116 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client_addr": "172.21.2.116:0/2452094959",
, 2021-03-11T17:02:12.116 INFO:teuthology.orchestra.run.gibba029.stdout:                    "tid": 8518

Ignoring malparsed backtrace: 2021-03-11T17:02:12.142 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:12.142 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",
, 2021-03-11T17:02:12.143 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:33.075251+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:12.168 INFO:teuthology.orchestra.run.gibba029.stdout:                ]
, 2021-03-11T17:02:12.168 INFO:teuthology.orchestra.run.gibba029.stdout:            }
, 2021-03-11T17:02:12.169 INFO:teuthology.orchestra.run.gibba029.stdout:        }

Ignoring malparsed backtrace: 2021-03-11T17:02:22.974+0000 7ffb03226700 -1 mon.a@0(leader).osd e278 definitely_dead 0
, marked in osd.6.
, 2021-03-11T17:02:23.078 INFO:tasks.thrashosds.thrasher:Added osd 6

Ignoring malparsed backtrace: 2021-03-11T17:02:41.793 INFO:teuthology.orchestra.run.gibba016.stdout:{"epoch":297,"fsid":"0288d25f-2798-4486-b9ed-6c1fa538c347","created":"2021-03-11T16:57:25.062737+0000","modified":"2021-03-11T17:02:41.330526+0000","last_up_change":"2021-03-11T17:02:19.040478+0000","last_in_change":"2021-03-11T17:02:23.058191+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T16:57:34.118119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"280","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"176","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T16:57:35.448919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":48,"pg_placement_num":48,"pg_placement_num_target":48,"pg_num_target":48,"pg_num_pending":48,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"142","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T16:57:46.854374+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"281","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"236","auid":0,"snap_mode":"selfmanaged","snap_seq":219,"snap_epoch":273,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ba2f09a4-7d62-494a-bc9a-b7dfc6d60bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":152,"up_from":187,"up_thru":295,"down_at":153,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6800","nonce":58253},{"type":"v1","addr":"172.21.2.116:6801","nonce":58253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6802","nonce":58253},{"type":"v1","addr":"172.21.2.116:6803","nonce":58253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6806","nonce":58253},{"type":"v1","addr":"172.21.2.116:6807","nonce":58253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6804","nonce":58253},{"type":"v1","addr":"172.21.2.116:6805","nonce":58253}]},"public_addr":"172.21.2.116:6801/58253","cluster_addr":"172.21.2.116:6803/58253","heartbeat_back_addr":"172.21.2.116:6807/58253","heartbeat_front_addr":"172.21.2.116:6805/58253","state":["exists","up"]},{"osd":1,"uuid":"577613bb-a808-4a5f-80d4-d71d0bb35676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":270,"up_from":277,"up_thru":291,"down_at":271,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6809","nonce":69535},{"type":"v1","addr":"172.21.2.116:6811","nonce":69535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6813","nonce":69535},{"type":"v1","addr":"172.21.2.116:6815","nonce":69535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6821","nonce":69535},{"type":"v1","addr":"172.21.2.116:6824","nonce":69535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6817","nonce":69535},{"type":"v1","addr":"172.21.2.116:6819","nonce":69535}]},"public_addr":"172.21.2.116:6811/69535","cluster_addr":"172.21.2.116:6815/69535","heartbeat_back_addr":"172.21.2.116:6824/69535","heartbeat_front_addr":"172.21.2.116:6819/69535","state":["exists","up"]},{"osd":2,"uuid":"c8fd4d3a-73f2-484d-8d9a-4bcc168398bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6808","nonce":35126},{"type":"v1","addr":"172.21.2.116:6810","nonce":35126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6812","nonce":35126},{"type":"v1","addr":"172.21.2.116:6814","nonce":35126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6820","nonce":35126},{"type":"v1","addr":"172.21.2.116:6823","nonce":35126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6816","nonce":35126},{"type":"v1","addr":"172.21.2.116:6818","nonce":35126}]},"public_addr":"172.21.2.116:6810/35126","cluster_addr":"172.21.2.116:6814/35126","heartbeat_back_addr":"172.21.2.116:6823/35126","heartbeat_front_addr":"172.21.2.116:6818/35126","state":["exists","up"]},{"osd":3,"uuid":"e0437b6a-c36d-4401-9cc0-fc64443fb0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":292,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6822","nonce":35127},{"type":"v1","addr":"172.21.2.116:6825","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6826","nonce":35127},{"type":"v1","addr":"172.21.2.116:6827","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6830","nonce":35127},{"type":"v1","addr":"172.21.2.116:6831","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6828","nonce":35127},{"type":"v1","addr":"172.21.2.116:6829","nonce":35127}]},"public_addr":"172.21.2.116:6825/35127","cluster_addr":"172.21.2.116:6827/35127","heartbeat_back_addr":"172.21.2.116:6831/35127","heartbeat_front_addr":"172.21.2.116:6829/35127","state":["exists","up"]},{"osd":4,"uuid":"adb93d6b-fdd1-4cf2-a670-0b579ff1556d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":293,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6800","nonce":35136},{"type":"v1","addr":"172.21.2.129:6801","nonce":35136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6802","nonce":35136},{"type":"v1","addr":"172.21.2.129:6803","nonce":35136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6806","nonce":35136},{"type":"v1","addr":"172.21.2.129:6807","nonce":35136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6804","nonce":35136},{"type":"v1","addr":"172.21.2.129:6805","nonce":35136}]},"public_addr":"172.21.2.129:6801/35136","cluster_addr":"172.21.2.129:6803/35136","heartbeat_back_addr":"172.21.2.129:6807/35136","heartbeat_front_addr":"172.21.2.129:6805/35136","state":["exists","up"]},{"osd":5,"uuid":"24747ad1-47a9-4a4a-b44b-8e95ac3dff7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":280,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6808","nonce":35137},{"type":"v1","addr":"172.21.2.129:6809","nonce":35137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6810","nonce":35137},{"type":"v1","addr":"172.21.2.129:6811","nonce":35137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6814","nonce":35137},{"type":"v1","addr":"172.21.2.129:6815","nonce":35137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6812","nonce":35137},{"type":"v1","addr":"172.21.2.129:6813","nonce":35137}]},"public_addr":"172.21.2.129:6809/35137","cluster_addr":"172.21.2.129:6811/35137","heartbeat_back_addr":"172.21.2.129:6815/35137","heartbeat_front_addr":"172.21.2.129:6813/35137","state":["exists","up"]},{"osd":6,"uuid":"d22cf1cd-03e3-4a34-9371-88c2fedca9a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":125,"up_from":136,"up_thru":294,"down_at":126,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6817","nonce":44841},{"type":"v1","addr":"172.21.2.129:6819","nonce":44841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6821","nonce":44841},{"type":"v1","addr":"172.21.2.129:6823","nonce":44841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6830","nonce":44841},{"type":"v1","addr":"172.21.2.129:6831","nonce":44841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6825","nonce":44841},{"type":"v1","addr":"172.21.2.129:6828","nonce":44841}]},"public_addr":"172.21.2.129:6819/44841","cluster_addr":"172.21.2.129:6823/44841","heartbeat_back_addr":"172.21.2.129:6831/44841","heartbeat_front_addr":"172.21.2.129:6828/44841","state":["exists","up"]},{"osd":7,"uuid":"acf539ae-1f07-403d-bb98-7b6ec25f2a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":288,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6816","nonce":35139},{"type":"v1","addr":"172.21.2.129:6818","nonce":35139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6820","nonce":35139},{"type":"v1","addr":"172.21.2.129:6822","nonce":35139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6827","nonce":35139},{"type":"v1","addr":"172.21.2.129:6829","nonce":35139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6824","nonce":35139},{"type":"v1","addr":"172.21.2.129:6826","nonce":35139}]},"public_addr":"172.21.2.129:6818/35139","cluster_addr":"172.21.2.129:6822/35139","heartbeat_back_addr":"172.21.2.129:6829/35139","heartbeat_front_addr":"172.21.2.129:6826/35139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:01:44.146927+0000","dead_epoch":153},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:33.859749+0000","dead_epoch":271},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.108809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.074493+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:33.962549+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.620024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.155841+0000","dead_epoch":126},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.574020+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":1,"to":2}]}],"pg_temp":[{"pgid":"3.c","osds":[5,3,2147483647]}],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":140,"length":1},{"begin":147,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:02:41.804 DEBUG:teuthology.orchestra.run.gibba016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats
, 2021-03-11T17:02:41.894+0000 7f8adceb1700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:02:48.019 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout
, 2021-03-11T17:02:48.020 DEBUG:teuthology.orchestra.run.gibba016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-03-11T17:02:48.110+0000 7f57b5c48700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:02:55.830 INFO:tasks.ceph:Scrubbing osd.6
, 2021-03-11T17:02:55.830 DEBUG:teuthology.orchestra.run.gibba016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:02:55.918+0000 7f1251d00700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:02:56.822 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2021-03-11T17:02:14.286275+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=14, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:02:56.823 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2021-03-11T17:02:48.781539+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=48, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:02:56.823 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2021-03-11T17:00:31.204784+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=0, tm_sec=31, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:03:37.397 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2021-03-11T17:01:44.368297+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=1, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:03:37.398 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2021-03-11T17:02:52.191793+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:03:37.399 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2021-03-11T17:01:22.260303+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=1, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:04:18.791 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-11T17:04:18.798 INFO:tasks.ceph.mon.b:Stopped
, 2021-03-11T17:04:18.799 INFO:tasks.ceph:Checking cluster log for badness...

Ignoring malparsed backtrace: 2021-03-11T17:05:14.992 INFO:teuthology.orchestra.run.gibba016.stdout:^* hv02.front.sepia.ceph.com     2   6   377    16   +613us[ +615us] +/-   30ms
, 2021-03-11T17:05:14.992 INFO:teuthology.orchestra.run.gibba016.stdout:^+ hv03.front.sepia.ceph.com     3   6   377    16  -1656us[-1656us] +/-   76ms
, 2021-03-11T17:05:14.992 INFO:teuthology.orchestra.run.gibba016.stdout:^+ hv04.front.sepia.ceph.com     3   6   377    15   +206us[ +206us] +/-   71ms

Ignoring malparsed backtrace:  ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f66d3dda636]
,  2: /usr/lib64/ceph/libceph-common.so.2(+0x277850) [0x7f66d3dda850]

Ignoring malparsed backtrace: 2021-03-11T17:02:09.688 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.688 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",
, 2021-03-11T17:02:09.688 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:35.080424+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.716 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.717 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.718 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.741 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.9728000000000003e-05
, 2021-03-11T17:02:09.741 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.741 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:09.785 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:35.702201+0000",
, 2021-03-11T17:02:09.785 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.3297999999999999e-05
, 2021-03-11T17:02:09.785 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:09.808 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",
, 2021-03-11T17:02:09.808 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:35.903609+0000",
, 2021-03-11T17:02:09.808 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.7372000000000001e-05

Ignoring malparsed backtrace: 2021-03-11T17:02:09.830 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.830 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",
, 2021-03-11T17:02:09.830 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:36.113531+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.854 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.854 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:09.854 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:02:09.877 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:09.877 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:09.877 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:09.900 INFO:teuthology.orchestra.run.gibba029.stdout:                    "tid": 7734
, 2021-03-11T17:02:09.900 INFO:teuthology.orchestra.run.gibba029.stdout:                },
, 2021-03-11T17:02:09.901 INFO:teuthology.orchestra.run.gibba029.stdout:                "events": [

Ignoring malparsed backtrace: 2021-03-11T17:02:09.922 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:44.291735+0000",
, 2021-03-11T17:02:09.922 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 2.1678000000000001e-05
, 2021-03-11T17:02:09.922 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:09.948 INFO:teuthology.orchestra.run.gibba029.stdout:        }
, 2021-03-11T17:02:09.948 INFO:teuthology.orchestra.run.gibba029.stdout:    ]
, 2021-03-11T17:02:09.948 INFO:teuthology.orchestra.run.gibba029.stdout:}

Ignoring malparsed backtrace: 2021-03-11T17:02:10.327 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:58:26.875758+0000",
, 2021-03-11T17:02:10.327 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:10.327 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:10.351 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:58:26.914304+0000",
, 2021-03-11T17:02:10.351 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.038483092000000003
, 2021-03-11T17:02:10.351 INFO:teuthology.orchestra.run.gibba029.stdout:                    }

Ignoring malparsed backtrace: 2021-03-11T17:02:10.375 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",
, 2021-03-11T17:02:10.376 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:10.908086+0000",
, 2021-03-11T17:02:10.376 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 2.7968999999999999e-05

Ignoring malparsed backtrace: 2021-03-11T17:02:10.399 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "started",
, 2021-03-11T17:02:10.399 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:10.914856+0000",
, 2021-03-11T17:02:10.399 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.024848762999999999

Ignoring malparsed backtrace: 2021-03-11T17:02:10.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "sub_op_committed",
, 2021-03-11T17:02:10.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:41.138198+0000",
, 2021-03-11T17:02:10.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.0059484389999999998

Ignoring malparsed backtrace: 2021-03-11T17:02:10.448 INFO:teuthology.orchestra.run.gibba029.stdout:            }
, 2021-03-11T17:02:10.448 INFO:teuthology.orchestra.run.gibba029.stdout:        },
, 2021-03-11T17:02:10.448 INFO:teuthology.orchestra.run.gibba029.stdout:        {

Ignoring malparsed backtrace: 2021-03-11T17:02:10.470 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client": "client.4572",
, 2021-03-11T17:02:10.471 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client_addr": "172.21.2.116:0/2452094959",
, 2021-03-11T17:02:10.471 INFO:teuthology.orchestra.run.gibba029.stdout:                    "tid": 7571

Ignoring malparsed backtrace: 2021-03-11T17:02:10.493 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.493 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "initiated",
, 2021-03-11T17:02:10.494 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:32.179292+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.516 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.516 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "initiated",
, 2021-03-11T17:02:10.516 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:38.087316+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.538 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.539 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "waiting for missing object",
, 2021-03-11T17:02:10.539 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:38.088509+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.562 INFO:teuthology.orchestra.run.gibba029.stdout:                "client_info": {
, 2021-03-11T17:02:10.562 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client": "client.4572",
, 2021-03-11T17:02:10.562 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client_addr": "172.21.2.116:0/2452094959",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.584 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.584 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.584 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.606 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.606 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.607 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.628 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.629 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.629 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.651 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.651 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.651 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.673 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:10.673 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:10.673 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:10.930 INFO:tasks.thrashosds:joining thrashosds
, 2021-03-11T17:02:10.904+0000 7f94028cd700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5  (PID: 35128) UID: 0
, 2021-03-11T17:02:10.951 INFO:teuthology.orchestra.run.gibba029.stdout:{

Ignoring malparsed backtrace: 2021-03-11T17:02:11.127 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.127 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "initiated",
, 2021-03-11T17:02:11.127 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:58:35.943479+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.150 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.150 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",
, 2021-03-11T17:02:11.151 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:38.891562+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.175 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.175 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "waiting_for_map not empty",
, 2021-03-11T17:02:11.175 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:38.891726+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.200 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.201 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "started",
, 2021-03-11T17:02:11.201 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T16:59:39.231978+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.222 INFO:teuthology.orchestra.run.gibba029.stdout:                ]
, 2021-03-11T17:02:11.223 INFO:teuthology.orchestra.run.gibba029.stdout:            }
, 2021-03-11T17:02:11.223 INFO:teuthology.orchestra.run.gibba029.stdout:        },

Ignoring malparsed backtrace: 2021-03-11T17:02:11.245 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.245 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.245 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.268 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.268 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.268 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.291 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.291 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.291 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.312 INFO:teuthology.orchestra.run.gibba029.stdout:            "description": "osd_op(client.4572.0:8599 3.fs0 3:f1e85c40:::gibba01641143-94:head [write 1859448~425558] snapc ba=[ba,b8,b5,b3,b2,ac,a7,a3,9c,9b,93,8c] ondisk+write+known_if_redirected e241)",
, 2021-03-11T17:02:11.313 INFO:teuthology.orchestra.run.gibba029.stdout:            "initiated_at": "2021-03-11T17:01:38.027124+0000",
, 2021-03-11T17:02:11.313 INFO:teuthology.orchestra.run.gibba029.stdout:            "age": 33.077382381,

Ignoring malparsed backtrace: 2021-03-11T17:02:11.334 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:11.335 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.335 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.356 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 9.5000000000000005e-06
, 2021-03-11T17:02:11.357 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.357 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.378 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00012094700000000001
, 2021-03-11T17:02:11.378 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.378 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.401 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 8.3714000000000005e-05
, 2021-03-11T17:02:11.401 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.401 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.425 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00040583399999999999
, 2021-03-11T17:02:11.425 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.425 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.447 INFO:teuthology.orchestra.run.gibba029.stdout:        {
, 2021-03-11T17:02:11.447 INFO:teuthology.orchestra.run.gibba029.stdout:            "description": "osd_op(client.4572.0:9245 3.es0 3:71948077:::gibba01641143-693:head [copy-from ver 206] snapc d7=[d7,d5,d1,d0,ce,c7,c2,93] ondisk+write+known_if_redirected e270)",
, 2021-03-11T17:02:11.448 INFO:teuthology.orchestra.run.gibba029.stdout:            "initiated_at": "2021-03-11T17:02:06.907966+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.470 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 2.0030282439999998
, 2021-03-11T17:02:11.470 INFO:teuthology.orchestra.run.gibba029.stdout:                    }
, 2021-03-11T17:02:11.470 INFO:teuthology.orchestra.run.gibba029.stdout:                ]

Ignoring malparsed backtrace: 2021-03-11T17:02:11.592 INFO:teuthology.orchestra.run.gibba029.stdout:    "num_ops": 0
, 2021-03-11T17:02:11.592 INFO:teuthology.orchestra.run.gibba029.stdout:}
, 2021-03-11T17:02:11.598 DEBUG:teuthology.orchestra.run.gibba029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops

Ignoring malparsed backtrace: 2021-03-11T17:02:11.822 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.822 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.822 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.845 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.845 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:11.845 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-03-11T17:02:11.868 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 3.3759999999999999e-06
, 2021-03-11T17:02:11.868 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.868 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.890 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00075816700000000002
, 2021-03-11T17:02:11.890 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.891 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.912 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 4.2586000000000002e-05
, 2021-03-11T17:02:11.913 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.913 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.936 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00071467500000000001
, 2021-03-11T17:02:11.936 INFO:teuthology.orchestra.run.gibba029.stdout:                    },
, 2021-03-11T17:02:11.936 INFO:teuthology.orchestra.run.gibba029.stdout:                    {

Ignoring malparsed backtrace: 2021-03-11T17:02:11.963 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:00:59.900788+0000",
, 2021-03-11T17:02:11.963 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0
, 2021-03-11T17:02:11.963 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:11.985 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:07.937528+0000",
, 2021-03-11T17:02:11.986 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 4.8640000000000004e-06
, 2021-03-11T17:02:11.986 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:12.009 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:07.940262+0000",
, 2021-03-11T17:02:12.010 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.0024267970000000001
, 2021-03-11T17:02:12.010 INFO:teuthology.orchestra.run.gibba029.stdout:                    },

Ignoring malparsed backtrace: 2021-03-11T17:02:12.036 INFO:teuthology.orchestra.run.gibba029.stdout:            "duration": 0.0067861859999999996,
, 2021-03-11T17:02:12.036 INFO:teuthology.orchestra.run.gibba029.stdout:            "type_data": {
, 2021-03-11T17:02:12.036 INFO:teuthology.orchestra.run.gibba029.stdout:                "flag_point": "commit sent; apply or cleanup",

Ignoring malparsed backtrace: 2021-03-11T17:02:12.062 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "all_read",
, 2021-03-11T17:02:12.062 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:17.958750+0000",
, 2021-03-11T17:02:12.063 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 7.5850000000000002e-06

Ignoring malparsed backtrace: 2021-03-11T17:02:12.088 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "sub_op_started",
, 2021-03-11T17:02:12.088 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:17.966891+0000",
, 2021-03-11T17:02:12.089 INFO:teuthology.orchestra.run.gibba029.stdout:                        "duration": 0.00090775700000000003

Ignoring malparsed backtrace: 2021-03-11T17:02:12.115 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client": "client.4572",
, 2021-03-11T17:02:12.116 INFO:teuthology.orchestra.run.gibba029.stdout:                    "client_addr": "172.21.2.116:0/2452094959",
, 2021-03-11T17:02:12.116 INFO:teuthology.orchestra.run.gibba029.stdout:                    "tid": 8518

Ignoring malparsed backtrace: 2021-03-11T17:02:12.142 INFO:teuthology.orchestra.run.gibba029.stdout:                    {
, 2021-03-11T17:02:12.142 INFO:teuthology.orchestra.run.gibba029.stdout:                        "event": "dispatched",
, 2021-03-11T17:02:12.143 INFO:teuthology.orchestra.run.gibba029.stdout:                        "time": "2021-03-11T17:01:33.075251+0000",

Ignoring malparsed backtrace: 2021-03-11T17:02:12.168 INFO:teuthology.orchestra.run.gibba029.stdout:                ]
, 2021-03-11T17:02:12.168 INFO:teuthology.orchestra.run.gibba029.stdout:            }
, 2021-03-11T17:02:12.169 INFO:teuthology.orchestra.run.gibba029.stdout:        }

Ignoring malparsed backtrace: 2021-03-11T17:02:22.974+0000 7ffb03226700 -1 mon.a@0(leader).osd e278 definitely_dead 0
, marked in osd.6.
, 2021-03-11T17:02:23.078 INFO:tasks.thrashosds.thrasher:Added osd 6

Ignoring malparsed backtrace: 2021-03-11T17:02:41.793 INFO:teuthology.orchestra.run.gibba016.stdout:{"epoch":297,"fsid":"0288d25f-2798-4486-b9ed-6c1fa538c347","created":"2021-03-11T16:57:25.062737+0000","modified":"2021-03-11T17:02:41.330526+0000","last_up_change":"2021-03-11T17:02:19.040478+0000","last_in_change":"2021-03-11T17:02:23.058191+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-11T16:57:34.118119+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"280","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"176","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-11T16:57:35.448919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":48,"pg_placement_num":48,"pg_placement_num_target":48,"pg_num_target":48,"pg_num_pending":48,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"142","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-11T16:57:46.854374+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"281","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"236","auid":0,"snap_mode":"selfmanaged","snap_seq":219,"snap_epoch":273,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ba2f09a4-7d62-494a-bc9a-b7dfc6d60bb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":152,"up_from":187,"up_thru":295,"down_at":153,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6800","nonce":58253},{"type":"v1","addr":"172.21.2.116:6801","nonce":58253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6802","nonce":58253},{"type":"v1","addr":"172.21.2.116:6803","nonce":58253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6806","nonce":58253},{"type":"v1","addr":"172.21.2.116:6807","nonce":58253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6804","nonce":58253},{"type":"v1","addr":"172.21.2.116:6805","nonce":58253}]},"public_addr":"172.21.2.116:6801/58253","cluster_addr":"172.21.2.116:6803/58253","heartbeat_back_addr":"172.21.2.116:6807/58253","heartbeat_front_addr":"172.21.2.116:6805/58253","state":["exists","up"]},{"osd":1,"uuid":"577613bb-a808-4a5f-80d4-d71d0bb35676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":270,"up_from":277,"up_thru":291,"down_at":271,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6809","nonce":69535},{"type":"v1","addr":"172.21.2.116:6811","nonce":69535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6813","nonce":69535},{"type":"v1","addr":"172.21.2.116:6815","nonce":69535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6821","nonce":69535},{"type":"v1","addr":"172.21.2.116:6824","nonce":69535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6817","nonce":69535},{"type":"v1","addr":"172.21.2.116:6819","nonce":69535}]},"public_addr":"172.21.2.116:6811/69535","cluster_addr":"172.21.2.116:6815/69535","heartbeat_back_addr":"172.21.2.116:6824/69535","heartbeat_front_addr":"172.21.2.116:6819/69535","state":["exists","up"]},{"osd":2,"uuid":"c8fd4d3a-73f2-484d-8d9a-4bcc168398bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6808","nonce":35126},{"type":"v1","addr":"172.21.2.116:6810","nonce":35126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6812","nonce":35126},{"type":"v1","addr":"172.21.2.116:6814","nonce":35126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6820","nonce":35126},{"type":"v1","addr":"172.21.2.116:6823","nonce":35126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6816","nonce":35126},{"type":"v1","addr":"172.21.2.116:6818","nonce":35126}]},"public_addr":"172.21.2.116:6810/35126","cluster_addr":"172.21.2.116:6814/35126","heartbeat_back_addr":"172.21.2.116:6823/35126","heartbeat_front_addr":"172.21.2.116:6818/35126","state":["exists","up"]},{"osd":3,"uuid":"e0437b6a-c36d-4401-9cc0-fc64443fb0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":292,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6822","nonce":35127},{"type":"v1","addr":"172.21.2.116:6825","nonce":35127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6826","nonce":35127},{"type":"v1","addr":"172.21.2.116:6827","nonce":35127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6830","nonce":35127},{"type":"v1","addr":"172.21.2.116:6831","nonce":35127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.116:6828","nonce":35127},{"type":"v1","addr":"172.21.2.116:6829","nonce":35127}]},"public_addr":"172.21.2.116:6825/35127","cluster_addr":"172.21.2.116:6827/35127","heartbeat_back_addr":"172.21.2.116:6831/35127","heartbeat_front_addr":"172.21.2.116:6829/35127","state":["exists","up"]},{"osd":4,"uuid":"adb93d6b-fdd1-4cf2-a670-0b579ff1556d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":293,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6800","nonce":35136},{"type":"v1","addr":"172.21.2.129:6801","nonce":35136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6802","nonce":35136},{"type":"v1","addr":"172.21.2.129:6803","nonce":35136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6806","nonce":35136},{"type":"v1","addr":"172.21.2.129:6807","nonce":35136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6804","nonce":35136},{"type":"v1","addr":"172.21.2.129:6805","nonce":35136}]},"public_addr":"172.21.2.129:6801/35136","cluster_addr":"172.21.2.129:6803/35136","heartbeat_back_addr":"172.21.2.129:6807/35136","heartbeat_front_addr":"172.21.2.129:6805/35136","state":["exists","up"]},{"osd":5,"uuid":"24747ad1-47a9-4a4a-b44b-8e95ac3dff7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":280,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6808","nonce":35137},{"type":"v1","addr":"172.21.2.129:6809","nonce":35137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6810","nonce":35137},{"type":"v1","addr":"172.21.2.129:6811","nonce":35137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6814","nonce":35137},{"type":"v1","addr":"172.21.2.129:6815","nonce":35137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6812","nonce":35137},{"type":"v1","addr":"172.21.2.129:6813","nonce":35137}]},"public_addr":"172.21.2.129:6809/35137","cluster_addr":"172.21.2.129:6811/35137","heartbeat_back_addr":"172.21.2.129:6815/35137","heartbeat_front_addr":"172.21.2.129:6813/35137","state":["exists","up"]},{"osd":6,"uuid":"d22cf1cd-03e3-4a34-9371-88c2fedca9a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":125,"up_from":136,"up_thru":294,"down_at":126,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6817","nonce":44841},{"type":"v1","addr":"172.21.2.129:6819","nonce":44841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6821","nonce":44841},{"type":"v1","addr":"172.21.2.129:6823","nonce":44841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6830","nonce":44841},{"type":"v1","addr":"172.21.2.129:6831","nonce":44841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6825","nonce":44841},{"type":"v1","addr":"172.21.2.129:6828","nonce":44841}]},"public_addr":"172.21.2.129:6819/44841","cluster_addr":"172.21.2.129:6823/44841","heartbeat_back_addr":"172.21.2.129:6831/44841","heartbeat_front_addr":"172.21.2.129:6828/44841","state":["exists","up"]},{"osd":7,"uuid":"acf539ae-1f07-403d-bb98-7b6ec25f2a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":288,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6816","nonce":35139},{"type":"v1","addr":"172.21.2.129:6818","nonce":35139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6820","nonce":35139},{"type":"v1","addr":"172.21.2.129:6822","nonce":35139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6827","nonce":35139},{"type":"v1","addr":"172.21.2.129:6829","nonce":35139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6824","nonce":35139},{"type":"v1","addr":"172.21.2.129:6826","nonce":35139}]},"public_addr":"172.21.2.129:6818/35139","cluster_addr":"172.21.2.129:6822/35139","heartbeat_back_addr":"172.21.2.129:6829/35139","heartbeat_front_addr":"172.21.2.129:6826/35139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:01:44.146927+0000","dead_epoch":153},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:33.859749+0000","dead_epoch":271},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.108809+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.074493+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:33.962549+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.620024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.155841+0000","dead_epoch":126},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-11T17:02:34.574020+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":1,"to":2}]}],"pg_temp":[{"pgid":"3.c","osds":[5,3,2147483647]}],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":140,"length":1},{"begin":147,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2021-03-11T17:02:41.804 DEBUG:teuthology.orchestra.run.gibba016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats
, 2021-03-11T17:02:41.894+0000 7f8adceb1700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:02:48.019 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout
, 2021-03-11T17:02:48.020 DEBUG:teuthology.orchestra.run.gibba016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-03-11T17:02:48.110+0000 7f57b5c48700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:02:55.830 INFO:tasks.ceph:Scrubbing osd.6
, 2021-03-11T17:02:55.830 DEBUG:teuthology.orchestra.run.gibba016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0
, 2021-03-11T17:02:55.918+0000 7f1251d00700 -1 WARNING: all dangerous and experimental features are enabled.

Ignoring malparsed backtrace: 2021-03-11T17:02:56.822 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2021-03-11T17:02:14.286275+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=14, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:02:56.823 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2021-03-11T17:02:48.781539+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=48, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:02:56.823 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2021-03-11T17:00:31.204784+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=0, tm_sec=31, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:03:37.397 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2021-03-11T17:01:44.368297+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=1, tm_sec=44, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:03:37.398 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2021-03-11T17:02:52.191793+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)
, 2021-03-11T17:03:37.399 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2021-03-11T17:01:22.260303+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=1, tm_sec=22, tm_wday=3, tm_yday=70, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=11, tm_hour=17, tm_min=2, tm_sec=52, tm_wday=3, tm_yday=70, tm_isdst=0)

Ignoring malparsed backtrace: 2021-03-11T17:04:18.791 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-11T17:04:18.798 INFO:tasks.ceph.mon.b:Stopped
, 2021-03-11T17:04:18.799 INFO:tasks.ceph:Checking cluster log for badness...

Ignoring malparsed backtrace: 2021-03-11T17:05:14.992 INFO:teuthology.orchestra.run.gibba016.stdout:^* hv02.front.sepia.ceph.com     2   6   377    16   +613us[ +615us] +/-   30ms
, 2021-03-11T17:05:14.992 INFO:teuthology.orchestra.run.gibba016.stdout:^+ hv03.front.sepia.ceph.com     3   6   377    16  -1656us[-1656us] +/-   76ms
, 2021-03-11T17:05:14.992 INFO:teuthology.orchestra.run.gibba016.stdout:^+ hv04.front.sepia.ceph.com     3   6   377    15   +206us[ +206us] +/-   71ms

Found 8 distinct failure reasons
Failure: Command failed (workunit test cephadm/create_iscsi_disks.sh) on gibba044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=54c3c60e724db8935de8cae1fa5d6210af4a2cc6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cephadm/create_iscsi_disks.sh'
3 jobs: ['5956165', '5956192', '5956171']
suites intersection: ['task/test_e2e}']
suites union: ['rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable', 'rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing', 'task/test_e2e}']

Failure: Command failed on gibba033 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec6e8472-8289-11eb-9083-001a4aab830c -- ceph orch apply iscsi iscsi user password --placement '1;gibba033=iscsi.a'"
9 jobs: ['5956166', '5956185', '5956177', '5956168', '5956163', '5956173', '5956180', '5956160', '5956189']
suites intersection: []
suites union: ['1-tasks', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro1$/{ubuntu_20.04_kubic_testing}', 'distro1$/{ubuntu_20.04}', 'fixed-2', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']

Dead: None
3 jobs: ['5956161', '5956182', '5956194']

Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0'
2 jobs: ['5956183', '5956167']
suites intersection: ['ceph', 'clusters/{fixed-2', 'mon_election/connectivity', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'supported-random-distro$/{rhel_8}', 'thrashosds-health', 'workloads/ec-small-objects-overwrites}']
suites union: ['ceph', 'clusters/{fixed-2', 'fast/fast', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/fastread', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-small-objects-overwrites}']

Failure: Command failed on gibba044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:54c3c60e724db8935de8cae1fa5d6210af4a2cc6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1cd5a066-828d-11eb-9083-001a4aab830c -- ceph orch host ls --format=json'
2 jobs: ['5956178', '5956187']
suites intersection: ['fixed-2', 'msgr/async-v2only']
suites union: ['1-start', '2-thrash', '3-tasks/snaps-few-objects', 'fixed-2', 'mode/root', 'mon_election/classic', 'msgr/async-v2only', 'rados/cephadm/thrash/{0-distro/ubuntu_20.04_kubic_testing', 'rados/cephadm/with-work/{0-distro/ubuntu_20.04_kubic_stable', 'root}', 'start', 'tasks/rados_python}']

Crash: Test failure: test_pool_removal (tasks.mgr.test_progress.TestProgress)
ceph version 17.0.0-1956-g54c3c60e (54c3c60e724db8935de8cae1fa5d6210af4a2cc6) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7f23c793cb20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x55857f27d549]
 5: (KernelDevice::_aio_thread()+0x1254) [0x55857fdba214]
 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55857fdc55a1]
 7: (Thread::_entry_func(void*)+0xd) [0x55857f9f7abd]
 8: /lib64/libpthread.so.0(+0x814a) [0x7f23c793214a]
 9: clone()
1 jobs: ['5956175']
suites: ['debug/mgr', 'mon_election/classic', 'objectstore/bluestore-stupid', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_8}', 'tasks/progress}']

Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0'
2 jobs: ['5956195', '5956181']
suites intersection: ['mon_election/connectivity', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'thrashosds-health']
suites union: ['ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'thrashers/careful', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}']

Failure: Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
1 jobs: ['5956186']
suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_20.04_kubic_stable', 'task/test_orch_cli}']