Found 446 jobs
Ignoring malparsed backtrace:  ceph version 16.0.0-8560-g2ec13aff (2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f75bc004a2b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55c508fdec4c]

Ignoring malparsed backtrace: 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:30.353478+0000",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7710000000000002e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:40.902160+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "sub_op_committed",
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:41.517725+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.8519999999999999e-05
, 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.236 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00019351399999999999
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.986817+0000",
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.4547e-05
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991037+0000",
, 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7884999999999999e-05
, 2020-12-23T09:11:00.304 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.326 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991175+0000",
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.350 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "osd_op(client.4397.0:3669 3.1s0 3:87be181d:::smithi07319897-17:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected e59)",
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:36.991432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.48914957599999997
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.398 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00013705799999999999
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.1s0 72/62 ECSubWriteReply(tid=1041, last_complete=72'234, committed=1, applied=1))",
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:53.333432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:19.156648+0000",
, 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00069930899999999995
, 2020-12-23T09:11:00.446 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:08:30.652551+0000",
, 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 150.21913628300001,
, 2020-12-23T09:11:00.877 INFO:teuthology.orchestra.run.smithi032.stdout:            "duration": 0.467359252,

Ignoring malparsed backtrace: 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 3.0402000000000001e-05
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.913695+0000",
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00075951899999999995
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916025+0000",
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0389999999999999e-06
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.021 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916500+0000",
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.75637134699999997
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.043 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:36.683977+0000",
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00014753100000000001
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "throttled",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.164706+0000",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00056430599999999999

Ignoring malparsed backtrace: 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "reached_pg",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:39.183677+0000",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 4.0210000000000003e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.167477+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.155 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.bs0 113/64 ECSubWriteReply(tid=1062, last_complete=113'756, committed=1, applied=1))",
, 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:10:59.183418+0000",
, 2020-12-23T09:11:01.176 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 1.688269558,

Ignoring malparsed backtrace: 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "queued_for_pg",
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:02.412496+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.641 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00039126200000000002
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "started",
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:24.975873+0000",
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 8.8382999999999997e-05
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "client_addr": "172.21.15.73:0/4281828164",
, 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "tid": 7389
, 2020-12-23T09:11:01.724 INFO:teuthology.orchestra.run.smithi032.stdout:                },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:38.092756+0000",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0360000000000001e-06

Ignoring malparsed backtrace: 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        },
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.784 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.826 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.28262363899999998
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:26.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4
, 2020-12-23T09:11:26.870 INFO:teuthology.orchestra.run.smithi073.stdout:330712481810
, 2020-12-23T09:11:26.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 330712481811 got 330712481810 for osd.4

Ignoring malparsed backtrace: 2020-12-23T09:11:36.584 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-12-23T09:10:32.809191+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=32, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-12-23T09:10:34.468408+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=34, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2020-12-23T09:10:21.113097+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=21, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)

Ignoring malparsed backtrace: 2020-12-23T09:12:23.991 INFO:tasks.ceph.osd.3:Stopped
, 2020-12-23T09:12:23.993 DEBUG:tasks.ceph.osd.4:waiting for process to exit
, 2020-12-23T09:12:23.994 INFO:teuthology.orchestra.run:waiting for 300

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-12-23-1139/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8560-g2ec13aff (2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f75bc004a2b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55c508fdec4c]

Ignoring malparsed backtrace: 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:30.353478+0000",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7710000000000002e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:40.902160+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "sub_op_committed",
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:41.517725+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.8519999999999999e-05
, 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.236 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00019351399999999999
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.986817+0000",
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.4547e-05
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991037+0000",
, 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7884999999999999e-05
, 2020-12-23T09:11:00.304 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.326 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991175+0000",
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.350 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "osd_op(client.4397.0:3669 3.1s0 3:87be181d:::smithi07319897-17:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected e59)",
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:36.991432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.48914957599999997
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.398 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00013705799999999999
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.1s0 72/62 ECSubWriteReply(tid=1041, last_complete=72'234, committed=1, applied=1))",
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:53.333432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:19.156648+0000",
, 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00069930899999999995
, 2020-12-23T09:11:00.446 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:08:30.652551+0000",
, 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 150.21913628300001,
, 2020-12-23T09:11:00.877 INFO:teuthology.orchestra.run.smithi032.stdout:            "duration": 0.467359252,

Ignoring malparsed backtrace: 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 3.0402000000000001e-05
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.913695+0000",
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00075951899999999995
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916025+0000",
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0389999999999999e-06
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.021 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916500+0000",
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.75637134699999997
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.043 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:36.683977+0000",
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00014753100000000001
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "throttled",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.164706+0000",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00056430599999999999

Ignoring malparsed backtrace: 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "reached_pg",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:39.183677+0000",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 4.0210000000000003e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.167477+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.155 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.bs0 113/64 ECSubWriteReply(tid=1062, last_complete=113'756, committed=1, applied=1))",
, 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:10:59.183418+0000",
, 2020-12-23T09:11:01.176 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 1.688269558,

Ignoring malparsed backtrace: 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "queued_for_pg",
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:02.412496+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.641 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00039126200000000002
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "started",
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:24.975873+0000",
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 8.8382999999999997e-05
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "client_addr": "172.21.15.73:0/4281828164",
, 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "tid": 7389
, 2020-12-23T09:11:01.724 INFO:teuthology.orchestra.run.smithi032.stdout:                },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:38.092756+0000",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0360000000000001e-06

Ignoring malparsed backtrace: 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        },
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.784 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.826 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.28262363899999998
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:26.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4
, 2020-12-23T09:11:26.870 INFO:teuthology.orchestra.run.smithi073.stdout:330712481810
, 2020-12-23T09:11:26.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 330712481811 got 330712481810 for osd.4

Ignoring malparsed backtrace: 2020-12-23T09:11:36.584 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-12-23T09:10:32.809191+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=32, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-12-23T09:10:34.468408+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=34, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2020-12-23T09:10:21.113097+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=21, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)

Ignoring malparsed backtrace: 2020-12-23T09:12:23.991 INFO:tasks.ceph.osd.3:Stopped
, 2020-12-23T09:12:23.993 DEBUG:tasks.ceph.osd.4:waiting for process to exit
, 2020-12-23T09:12:23.994 INFO:teuthology.orchestra.run:waiting for 300

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-12-23-1139/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8560-g2ec13aff (2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f75bc004a2b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55c508fdec4c]

Ignoring malparsed backtrace: 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:30.353478+0000",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7710000000000002e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:40.902160+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "sub_op_committed",
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:41.517725+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.8519999999999999e-05
, 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.236 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00019351399999999999
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.986817+0000",
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.4547e-05
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991037+0000",
, 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7884999999999999e-05
, 2020-12-23T09:11:00.304 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.326 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991175+0000",
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.350 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "osd_op(client.4397.0:3669 3.1s0 3:87be181d:::smithi07319897-17:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected e59)",
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:36.991432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.48914957599999997
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.398 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00013705799999999999
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.1s0 72/62 ECSubWriteReply(tid=1041, last_complete=72'234, committed=1, applied=1))",
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:53.333432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:19.156648+0000",
, 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00069930899999999995
, 2020-12-23T09:11:00.446 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:08:30.652551+0000",
, 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 150.21913628300001,
, 2020-12-23T09:11:00.877 INFO:teuthology.orchestra.run.smithi032.stdout:            "duration": 0.467359252,

Ignoring malparsed backtrace: 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 3.0402000000000001e-05
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.913695+0000",
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00075951899999999995
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916025+0000",
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0389999999999999e-06
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.021 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916500+0000",
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.75637134699999997
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.043 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:36.683977+0000",
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00014753100000000001
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "throttled",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.164706+0000",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00056430599999999999

Ignoring malparsed backtrace: 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "reached_pg",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:39.183677+0000",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 4.0210000000000003e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.167477+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.155 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.bs0 113/64 ECSubWriteReply(tid=1062, last_complete=113'756, committed=1, applied=1))",
, 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:10:59.183418+0000",
, 2020-12-23T09:11:01.176 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 1.688269558,

Ignoring malparsed backtrace: 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "queued_for_pg",
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:02.412496+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.641 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00039126200000000002
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "started",
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:24.975873+0000",
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 8.8382999999999997e-05
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "client_addr": "172.21.15.73:0/4281828164",
, 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "tid": 7389
, 2020-12-23T09:11:01.724 INFO:teuthology.orchestra.run.smithi032.stdout:                },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:38.092756+0000",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0360000000000001e-06

Ignoring malparsed backtrace: 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        },
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.784 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.826 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.28262363899999998
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:26.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4
, 2020-12-23T09:11:26.870 INFO:teuthology.orchestra.run.smithi073.stdout:330712481810
, 2020-12-23T09:11:26.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 330712481811 got 330712481810 for osd.4

Ignoring malparsed backtrace: 2020-12-23T09:11:36.584 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-12-23T09:10:32.809191+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=32, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-12-23T09:10:34.468408+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=34, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2020-12-23T09:10:21.113097+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=21, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)

Ignoring malparsed backtrace: 2020-12-23T09:12:23.991 INFO:tasks.ceph.osd.3:Stopped
, 2020-12-23T09:12:23.993 DEBUG:tasks.ceph.osd.4:waiting for process to exit
, 2020-12-23T09:12:23.994 INFO:teuthology.orchestra.run:waiting for 300

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-12-23-1139/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8560-g2ec13aff (2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f75bc004a2b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55c508fdec4c]

Ignoring malparsed backtrace: 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:30.353478+0000",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7710000000000002e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:40.902160+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "sub_op_committed",
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:41.517725+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.8519999999999999e-05
, 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.236 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00019351399999999999
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.986817+0000",
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.4547e-05
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991037+0000",
, 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7884999999999999e-05
, 2020-12-23T09:11:00.304 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.326 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991175+0000",
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.350 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "osd_op(client.4397.0:3669 3.1s0 3:87be181d:::smithi07319897-17:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected e59)",
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:36.991432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.48914957599999997
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.398 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00013705799999999999
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.1s0 72/62 ECSubWriteReply(tid=1041, last_complete=72'234, committed=1, applied=1))",
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:53.333432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:19.156648+0000",
, 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00069930899999999995
, 2020-12-23T09:11:00.446 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:08:30.652551+0000",
, 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 150.21913628300001,
, 2020-12-23T09:11:00.877 INFO:teuthology.orchestra.run.smithi032.stdout:            "duration": 0.467359252,

Ignoring malparsed backtrace: 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 3.0402000000000001e-05
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.913695+0000",
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00075951899999999995
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916025+0000",
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0389999999999999e-06
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.021 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916500+0000",
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.75637134699999997
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.043 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:36.683977+0000",
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00014753100000000001
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "throttled",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.164706+0000",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00056430599999999999

Ignoring malparsed backtrace: 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "reached_pg",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:39.183677+0000",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 4.0210000000000003e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.167477+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.155 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.bs0 113/64 ECSubWriteReply(tid=1062, last_complete=113'756, committed=1, applied=1))",
, 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:10:59.183418+0000",
, 2020-12-23T09:11:01.176 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 1.688269558,

Ignoring malparsed backtrace: 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "queued_for_pg",
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:02.412496+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.641 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00039126200000000002
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "started",
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:24.975873+0000",
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 8.8382999999999997e-05
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "client_addr": "172.21.15.73:0/4281828164",
, 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "tid": 7389
, 2020-12-23T09:11:01.724 INFO:teuthology.orchestra.run.smithi032.stdout:                },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:38.092756+0000",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0360000000000001e-06

Ignoring malparsed backtrace: 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        },
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.784 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.826 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.28262363899999998
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:26.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4
, 2020-12-23T09:11:26.870 INFO:teuthology.orchestra.run.smithi073.stdout:330712481810
, 2020-12-23T09:11:26.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 330712481811 got 330712481810 for osd.4

Ignoring malparsed backtrace: 2020-12-23T09:11:36.584 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-12-23T09:10:32.809191+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=32, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-12-23T09:10:34.468408+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=34, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2020-12-23T09:10:21.113097+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=21, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)

Ignoring malparsed backtrace: 2020-12-23T09:12:23.991 INFO:tasks.ceph.osd.3:Stopped
, 2020-12-23T09:12:23.993 DEBUG:tasks.ceph.osd.4:waiting for process to exit
, 2020-12-23T09:12:23.994 INFO:teuthology.orchestra.run:waiting for 300

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-12-23-1139/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8560-g2ec13aff (2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f75bc004a2b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55c508fdec4c]

Ignoring malparsed backtrace: 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:30.353478+0000",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7710000000000002e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:40.902160+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "sub_op_committed",
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:41.517725+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.8519999999999999e-05
, 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.236 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00019351399999999999
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.986817+0000",
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.4547e-05
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991037+0000",
, 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7884999999999999e-05
, 2020-12-23T09:11:00.304 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.326 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991175+0000",
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.350 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "osd_op(client.4397.0:3669 3.1s0 3:87be181d:::smithi07319897-17:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected e59)",
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:36.991432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.48914957599999997
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.398 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00013705799999999999
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.1s0 72/62 ECSubWriteReply(tid=1041, last_complete=72'234, committed=1, applied=1))",
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:53.333432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:19.156648+0000",
, 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00069930899999999995
, 2020-12-23T09:11:00.446 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:08:30.652551+0000",
, 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 150.21913628300001,
, 2020-12-23T09:11:00.877 INFO:teuthology.orchestra.run.smithi032.stdout:            "duration": 0.467359252,

Ignoring malparsed backtrace: 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 3.0402000000000001e-05
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.913695+0000",
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00075951899999999995
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916025+0000",
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0389999999999999e-06
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.021 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916500+0000",
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.75637134699999997
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.043 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:36.683977+0000",
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00014753100000000001
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "throttled",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.164706+0000",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00056430599999999999

Ignoring malparsed backtrace: 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "reached_pg",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:39.183677+0000",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 4.0210000000000003e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.167477+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.155 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.bs0 113/64 ECSubWriteReply(tid=1062, last_complete=113'756, committed=1, applied=1))",
, 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:10:59.183418+0000",
, 2020-12-23T09:11:01.176 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 1.688269558,

Ignoring malparsed backtrace: 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "queued_for_pg",
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:02.412496+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.641 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00039126200000000002
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "started",
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:24.975873+0000",
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 8.8382999999999997e-05
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "client_addr": "172.21.15.73:0/4281828164",
, 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "tid": 7389
, 2020-12-23T09:11:01.724 INFO:teuthology.orchestra.run.smithi032.stdout:                },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:38.092756+0000",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0360000000000001e-06

Ignoring malparsed backtrace: 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        },
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.784 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.826 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.28262363899999998
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:26.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4
, 2020-12-23T09:11:26.870 INFO:teuthology.orchestra.run.smithi073.stdout:330712481810
, 2020-12-23T09:11:26.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 330712481811 got 330712481810 for osd.4

Ignoring malparsed backtrace: 2020-12-23T09:11:36.584 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-12-23T09:10:32.809191+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=32, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-12-23T09:10:34.468408+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=34, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2020-12-23T09:10:21.113097+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=21, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)

Ignoring malparsed backtrace: 2020-12-23T09:12:23.991 INFO:tasks.ceph.osd.3:Stopped
, 2020-12-23T09:12:23.993 DEBUG:tasks.ceph.osd.4:waiting for process to exit
, 2020-12-23T09:12:23.994 INFO:teuthology.orchestra.run:waiting for 300

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-12-23-1139/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8560-g2ec13aff (2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f75bc004a2b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55c508fdec4c]

Ignoring malparsed backtrace: 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:30.353478+0000",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7710000000000002e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:40.902160+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "sub_op_committed",
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:41.517725+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.8519999999999999e-05
, 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.236 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00019351399999999999
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.986817+0000",
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.4547e-05
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991037+0000",
, 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7884999999999999e-05
, 2020-12-23T09:11:00.304 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.326 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991175+0000",
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.350 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "osd_op(client.4397.0:3669 3.1s0 3:87be181d:::smithi07319897-17:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected e59)",
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:36.991432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.48914957599999997
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.398 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00013705799999999999
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.1s0 72/62 ECSubWriteReply(tid=1041, last_complete=72'234, committed=1, applied=1))",
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:53.333432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:19.156648+0000",
, 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00069930899999999995
, 2020-12-23T09:11:00.446 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:08:30.652551+0000",
, 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 150.21913628300001,
, 2020-12-23T09:11:00.877 INFO:teuthology.orchestra.run.smithi032.stdout:            "duration": 0.467359252,

Ignoring malparsed backtrace: 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 3.0402000000000001e-05
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.913695+0000",
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00075951899999999995
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916025+0000",
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0389999999999999e-06
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.021 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916500+0000",
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.75637134699999997
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.043 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:36.683977+0000",
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00014753100000000001
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "throttled",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.164706+0000",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00056430599999999999

Ignoring malparsed backtrace: 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "reached_pg",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:39.183677+0000",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 4.0210000000000003e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.167477+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.155 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.bs0 113/64 ECSubWriteReply(tid=1062, last_complete=113'756, committed=1, applied=1))",
, 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:10:59.183418+0000",
, 2020-12-23T09:11:01.176 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 1.688269558,

Ignoring malparsed backtrace: 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "queued_for_pg",
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:02.412496+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.641 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00039126200000000002
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "started",
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:24.975873+0000",
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 8.8382999999999997e-05
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "client_addr": "172.21.15.73:0/4281828164",
, 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "tid": 7389
, 2020-12-23T09:11:01.724 INFO:teuthology.orchestra.run.smithi032.stdout:                },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:38.092756+0000",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0360000000000001e-06

Ignoring malparsed backtrace: 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        },
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.784 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.826 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.28262363899999998
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:26.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4
, 2020-12-23T09:11:26.870 INFO:teuthology.orchestra.run.smithi073.stdout:330712481810
, 2020-12-23T09:11:26.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 330712481811 got 330712481810 for osd.4

Ignoring malparsed backtrace: 2020-12-23T09:11:36.584 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-12-23T09:10:32.809191+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=32, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-12-23T09:10:34.468408+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=34, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2020-12-23T09:10:21.113097+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=21, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)

Ignoring malparsed backtrace: 2020-12-23T09:12:23.991 INFO:tasks.ceph.osd.3:Stopped
, 2020-12-23T09:12:23.993 DEBUG:tasks.ceph.osd.4:waiting for process to exit
, 2020-12-23T09:12:23.994 INFO:teuthology.orchestra.run:waiting for 300

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-12-23-1139/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8560-g2ec13aff (2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f75bc004a2b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55c508fdec4c]

Ignoring malparsed backtrace: 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:30.353478+0000",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7710000000000002e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:40.902160+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "sub_op_committed",
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:41.517725+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.8519999999999999e-05
, 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.236 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00019351399999999999
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.986817+0000",
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.4547e-05
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991037+0000",
, 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7884999999999999e-05
, 2020-12-23T09:11:00.304 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.326 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991175+0000",
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.350 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "osd_op(client.4397.0:3669 3.1s0 3:87be181d:::smithi07319897-17:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected e59)",
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:36.991432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.48914957599999997
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.398 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00013705799999999999
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.1s0 72/62 ECSubWriteReply(tid=1041, last_complete=72'234, committed=1, applied=1))",
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:53.333432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:19.156648+0000",
, 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00069930899999999995
, 2020-12-23T09:11:00.446 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:08:30.652551+0000",
, 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 150.21913628300001,
, 2020-12-23T09:11:00.877 INFO:teuthology.orchestra.run.smithi032.stdout:            "duration": 0.467359252,

Ignoring malparsed backtrace: 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 3.0402000000000001e-05
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.913695+0000",
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00075951899999999995
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916025+0000",
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0389999999999999e-06
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.021 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916500+0000",
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.75637134699999997
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.043 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:36.683977+0000",
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00014753100000000001
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "throttled",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.164706+0000",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00056430599999999999

Ignoring malparsed backtrace: 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "reached_pg",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:39.183677+0000",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 4.0210000000000003e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.167477+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.155 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.bs0 113/64 ECSubWriteReply(tid=1062, last_complete=113'756, committed=1, applied=1))",
, 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:10:59.183418+0000",
, 2020-12-23T09:11:01.176 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 1.688269558,

Ignoring malparsed backtrace: 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "queued_for_pg",
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:02.412496+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.641 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00039126200000000002
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "started",
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:24.975873+0000",
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 8.8382999999999997e-05
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "client_addr": "172.21.15.73:0/4281828164",
, 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "tid": 7389
, 2020-12-23T09:11:01.724 INFO:teuthology.orchestra.run.smithi032.stdout:                },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:38.092756+0000",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0360000000000001e-06

Ignoring malparsed backtrace: 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        },
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.784 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.826 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.28262363899999998
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:26.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4
, 2020-12-23T09:11:26.870 INFO:teuthology.orchestra.run.smithi073.stdout:330712481810
, 2020-12-23T09:11:26.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 330712481811 got 330712481810 for osd.4

Ignoring malparsed backtrace: 2020-12-23T09:11:36.584 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-12-23T09:10:32.809191+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=32, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-12-23T09:10:34.468408+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=34, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2020-12-23T09:10:21.113097+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=21, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)

Ignoring malparsed backtrace: 2020-12-23T09:12:23.991 INFO:tasks.ceph.osd.3:Stopped
, 2020-12-23T09:12:23.993 DEBUG:tasks.ceph.osd.4:waiting for process to exit
, 2020-12-23T09:12:23.994 INFO:teuthology.orchestra.run:waiting for 300

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-12-23-1139/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Ignoring malparsed backtrace:  ceph version 16.0.0-8560-g2ec13aff (2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f75bc004a2b]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x55c508fdec4c]

Ignoring malparsed backtrace: 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:30.353478+0000",
, 2020-12-23T09:11:00.119 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7710000000000002e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:00.143 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:40.902160+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "sub_op_committed",
, 2020-12-23T09:11:00.165 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:41.517725+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.189 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:00.212 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.8519999999999999e-05
, 2020-12-23T09:11:00.235 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.236 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00019351399999999999
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.258 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.986817+0000",
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.4547e-05
, 2020-12-23T09:11:00.281 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991037+0000",
, 2020-12-23T09:11:00.303 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 1.7884999999999999e-05
, 2020-12-23T09:11:00.304 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.326 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:36.991175+0000",
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0
, 2020-12-23T09:11:00.327 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.350 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "osd_op(client.4397.0:3669 3.1s0 3:87be181d:::smithi07319897-17:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected e59)",
, 2020-12-23T09:11:00.351 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:36.991432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.48914957599999997
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.374 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.398 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00013705799999999999
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.399 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:        {
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.1s0 72/62 ECSubWriteReply(tid=1041, last_complete=72'234, committed=1, applied=1))",
, 2020-12-23T09:11:00.422 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:09:53.333432+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:19.156648+0000",
, 2020-12-23T09:11:00.445 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00069930899999999995
, 2020-12-23T09:11:00.446 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:08:30.652551+0000",
, 2020-12-23T09:11:00.876 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 150.21913628300001,
, 2020-12-23T09:11:00.877 INFO:teuthology.orchestra.run.smithi032.stdout:            "duration": 0.467359252,

Ignoring malparsed backtrace: 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 3.0402000000000001e-05
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:00.927 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-23T09:11:00.951 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.913695+0000",
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00075951899999999995
, 2020-12-23T09:11:00.975 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916025+0000",
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0389999999999999e-06
, 2020-12-23T09:11:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.021 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:35.916500+0000",
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.75637134699999997
, 2020-12-23T09:11:01.022 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.043 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:08:36.683977+0000",
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00014753100000000001
, 2020-12-23T09:11:01.044 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "throttled",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.164706+0000",
, 2020-12-23T09:11:01.066 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00056430599999999999

Ignoring malparsed backtrace: 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "reached_pg",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:39.183677+0000",
, 2020-12-23T09:11:01.088 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 4.0210000000000003e-05

Ignoring malparsed backtrace: 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "header_read",
, 2020-12-23T09:11:01.110 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:09:38.167477+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.132 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.154 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.155 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "description": "MOSDECSubOpWriteReply(3.bs0 113/64 ECSubWriteReply(tid=1062, last_complete=113'756, committed=1, applied=1))",
, 2020-12-23T09:11:01.175 INFO:teuthology.orchestra.run.smithi032.stdout:            "initiated_at": "2020-12-23T09:10:59.183418+0000",
, 2020-12-23T09:11:01.176 INFO:teuthology.orchestra.run.smithi032.stdout:            "age": 1.688269558,

Ignoring malparsed backtrace: 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "queued_for_pg",
, 2020-12-23T09:11:01.597 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:02.412496+0000",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.619 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.641 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.00039126200000000002
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:            "type_data": {
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "flag_point": "started",
, 2020-12-23T09:11:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:                "client_info": {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:24.975873+0000",
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 8.8382999999999997e-05
, 2020-12-23T09:11:01.703 INFO:teuthology.orchestra.run.smithi032.stdout:                    },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "client_addr": "172.21.15.73:0/4281828164",
, 2020-12-23T09:11:01.723 INFO:teuthology.orchestra.run.smithi032.stdout:                    "tid": 7389
, 2020-12-23T09:11:01.724 INFO:teuthology.orchestra.run.smithi032.stdout:                },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "dispatched",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "time": "2020-12-23T09:10:38.092756+0000",
, 2020-12-23T09:11:01.744 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 7.0360000000000001e-06

Ignoring malparsed backtrace: 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        },
, 2020-12-23T09:11:01.764 INFO:teuthology.orchestra.run.smithi032.stdout:        {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.784 INFO:teuthology.orchestra.run.smithi032.stdout:                ]
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:            }
, 2020-12-23T09:11:01.785 INFO:teuthology.orchestra.run.smithi032.stdout:        },

Ignoring malparsed backtrace: 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                    {
, 2020-12-23T09:11:01.805 INFO:teuthology.orchestra.run.smithi032.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                },
, 2020-12-23T09:11:01.825 INFO:teuthology.orchestra.run.smithi032.stdout:                "events": [
, 2020-12-23T09:11:01.826 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                        "duration": 0.28262363899999998
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    },
, 2020-12-23T09:11:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:                    {

Ignoring malparsed backtrace: 2020-12-23T09:11:26.569 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4
, 2020-12-23T09:11:26.870 INFO:teuthology.orchestra.run.smithi073.stdout:330712481810
, 2020-12-23T09:11:26.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 330712481811 got 330712481810 for osd.4

Ignoring malparsed backtrace: 2020-12-23T09:11:36.584 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-12-23T09:10:32.809191+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=32, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-12-23T09:10:34.468408+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=34, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)
, 2020-12-23T09:11:36.585 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2020-12-23T09:10:21.113097+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=10, tm_sec=21, tm_wday=2, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=23, tm_hour=9, tm_min=11, tm_sec=31, tm_wday=2, tm_yday=358, tm_isdst=0)

Ignoring malparsed backtrace: 2020-12-23T09:12:23.991 INFO:tasks.ceph.osd.3:Stopped
, 2020-12-23T09:12:23.993 DEBUG:tasks.ceph.osd.4:waiting for process to exit
, 2020-12-23T09:12:23.994 INFO:teuthology.orchestra.run:waiting for 300

Ignoring malparsed backtrace:   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
,     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-12-23-1139/qa/tasks/rados.py", line 277, in task

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 173, in run_tasks
,     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__

Found 9 distinct failure reasons
Dead: b'2020-12-23T23:33:26.319 DEBUG:teuthology.exit:Finished running handlers'
15 jobs: ['5732739', '5732519', '5732713', '5732633', '5732368', '5732617', '5732625', '5732545', '5732446', '5732527', '5732440', '5732744', '5732535', '5732538', '5732703']
suites intersection: ['openstack}', 'rados']
suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'thrashosds-health}', 'validater/valgrind}', 'workloads/cache-snaps}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-small-objects-many-deletes}', 'workloads/pool-snaps-few-objects}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/small-objects-balanced}']

Failure: reached maximum tries (180) after waiting for 180 seconds
6 jobs: ['5732745', '5732637', '5732415', '5732470', '5732520', '5732596']
suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start']
suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start']

Failure: Command failed (workunit test cephtool/test.sh) on smithi136 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
2 jobs: ['5732680', '5732790']
suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}']
suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}']

Failure: psutil.NoSuchProcess process no longer exists (pid=20667)
1 jobs: ['5732796']
suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}']

Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2 jobs: ['5732402', '5732558']
suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']
suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']

Failure: "2020-12-23T09:38:56.380228+0000 osd.6 (osd.6) 139 : cluster [ERR] scrub 57.1f 57:fb29a7b2:test-rados-api-smithi050-41536-74::foo:24 : size 0 != clone_size 10" in cluster log
1 jobs: ['5732435']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/rados_api_tests}']

Timeout 120 running ceph --cluster ceph osd dump --format=json
1 jobs: ['5732531']
suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}']

Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 20 --max-in-flight 8 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 0 --op write 0 --op delete 20 --op append 5 --op write_excl 0 --op append_excl 5 --pool unique_pool_0'
1 jobs: ['5732371']
suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-small-objects-many-deletes}']

Crash: Found coredumps on ubuntu@smithi186.front.sepia.ceph.com
ceph version 16.0.0-8560-g2ec13aff (2ec13aff05dfff9dc716b3dd3fa03bbff2cb80a7) pacific (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f334a7f2980]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x55da93d9a1ec]
 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55da93d9a376]
 6: (PG::scrub(unsigned int, ThreadPool::TPHandle&)+0x195) [0x55da93ed3ff5]
 7: (ceph::osd::scheduler::PGScrub::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1a) [0x55da9408186a]
 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xcd5) [0x55da93e3f5f5]
 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55da944b0f3c]
 10: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55da944b41f0]
 11: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f334a7e76db]
 12: clone()
1 jobs: ['5732801']
suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/pg-autoscaler', 'supported-random-distro$/{ubuntu_latest}}']