Found 49 jobs
Ignoring malparsed backtrace:  ceph version 17.0.0-8969-gfedc95a0 (fedc95a0997bbff4576a21a2ec553b013abc1e5b) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7fae79cdd78e]
,  2: /usr/lib/ceph/libceph-common.so.2(+0x27a9a0) [0x7fae79cdd9a0]

Ignoring malparsed backtrace: 2021-11-13T18:26:26.300 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:26.300 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:54.066014+0000",
, 2021-11-13T18:26:26.300 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.20352450999999999

Ignoring malparsed backtrace: 2021-11-13T18:26:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",
, 2021-11-13T18:26:26.328 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.611296+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.353 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.353 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:26.353 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:24:14.889227+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.376 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.376 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "started",
, 2021-11-13T18:26:26.376 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:24:14.994873+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.398 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.399 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:26.399 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:24:15.200041+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.422 INFO:teuthology.orchestra.run.smithi157.stdout:                ]
, 2021-11-13T18:26:26.422 INFO:teuthology.orchestra.run.smithi157.stdout:            }
, 2021-11-13T18:26:26.422 INFO:teuthology.orchestra.run.smithi157.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:26.445 INFO:teuthology.orchestra.run.smithi157.stdout:            "initiated_at": "2021-11-13T18:24:15.900255+0000",
, 2021-11-13T18:26:26.445 INFO:teuthology.orchestra.run.smithi157.stdout:            "age": 130.37645276999999,
, 2021-11-13T18:26:26.446 INFO:teuthology.orchestra.run.smithi157.stdout:            "duration": 0.50350779300000004,

Ignoring malparsed backtrace: 2021-11-13T18:26:26.468 INFO:teuthology.orchestra.run.smithi157.stdout:                "client_info": {
, 2021-11-13T18:26:26.469 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client": "client.4439",
, 2021-11-13T18:26:26.469 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client_addr": "172.21.15.157:0/1116373644",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.491 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.491 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.491 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.514 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.514 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.514 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.537 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.537 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.537 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.560 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.560 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.560 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting_for_map not empty",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.583 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.583 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.583 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "commit_sent",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.605 INFO:teuthology.orchestra.run.smithi157.stdout:                    }
, 2021-11-13T18:26:26.605 INFO:teuthology.orchestra.run.smithi157.stdout:                ]
, 2021-11-13T18:26:26.606 INFO:teuthology.orchestra.run.smithi157.stdout:            }

Ignoring malparsed backtrace: 2021-11-13T18:26:26.628 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 1.0631999999999999e-05
, 2021-11-13T18:26:26.628 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.629 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:26.651 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.000107763
, 2021-11-13T18:26:26.651 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.651 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:26.816 INFO:teuthology.orchestra.run.smithi157.stdout:{
, 2021-11-13T18:26:26.817 INFO:teuthology.orchestra.run.smithi157.stdout:    "ops": [
, 2021-11-13T18:26:26.817 INFO:teuthology.orchestra.run.smithi157.stdout:        {

Ignoring malparsed backtrace: 2021-11-13T18:26:26.851 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.851 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:26.852 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:26:20.352153+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.878 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.878 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.878 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.105 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "op_commit",
, 2021-11-13T18:26:27.105 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:47.796640+0000",
, 2021-11-13T18:26:27.105 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.0046705610000000002

Ignoring malparsed backtrace: 2021-11-13T18:26:27.133 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "commit_sent",
, 2021-11-13T18:26:27.133 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:48.109866+0000",
, 2021-11-13T18:26:27.133 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 6.8777999999999998e-05

Ignoring malparsed backtrace: 2021-11-13T18:26:27.156 INFO:teuthology.orchestra.run.smithi157.stdout:            "age": 218.36948089699999,
, 2021-11-13T18:26:27.156 INFO:teuthology.orchestra.run.smithi157.stdout:            "duration": 0.83512053799999997,
, 2021-11-13T18:26:27.156 INFO:teuthology.orchestra.run.smithi157.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-11-13T18:26:27.178 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.178 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",
, 2021-11-13T18:26:27.178 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:49.653555+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.201 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.201 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "dispatched",
, 2021-11-13T18:26:27.201 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.067104+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.222 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.223 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:27.223 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.072206+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "dispatched",
, 2021-11-13T18:26:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.072310+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.265 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.265 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",
, 2021-11-13T18:26:27.266 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.072389+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.287 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.288 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:27.288 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.612311+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.376 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.376 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.376 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.397 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.397 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.397 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.418 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.418 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.418 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.441 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.441 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.441 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.463 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 4294967295.9999728
, 2021-11-13T18:26:27.463 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.463 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:27.619 INFO:teuthology.orchestra.run.smithi183.stdout:{
, 2021-11-13T18:26:27.619 INFO:teuthology.orchestra.run.smithi183.stdout:    "ops": [],
, 2021-11-13T18:26:27.620 INFO:teuthology.orchestra.run.smithi183.stdout:    "num_ops": 0

Ignoring malparsed backtrace: 2021-11-13T18:26:27.867 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:10.025302+0000",
, 2021-11-13T18:26:27.867 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00064953299999999999
, 2021-11-13T18:26:27.867 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:27.891 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:26.468508+0000",
, 2021-11-13T18:26:27.891 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.112490749
, 2021-11-13T18:26:27.891 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:27.918 INFO:teuthology.orchestra.run.smithi183.stdout:        },
, 2021-11-13T18:26:27.918 INFO:teuthology.orchestra.run.smithi183.stdout:        {
, 2021-11-13T18:26:27.919 INFO:teuthology.orchestra.run.smithi183.stdout:            "description": "osd_op(client.4439.0:4005 4.f 4:f78ab5fc:::smithi15722858-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.948 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "dispatched",
, 2021-11-13T18:26:27.948 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:32.063624+0000",
, 2021-11-13T18:26:27.949 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 3.4173e-05

Ignoring malparsed backtrace: 2021-11-13T18:26:27.974 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:27.974 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:00.279540+0000",
, 2021-11-13T18:26:27.974 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 2.5679999999999998e-06

Ignoring malparsed backtrace: 2021-11-13T18:26:28.002 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:28.002 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:00.686238+0000",
, 2021-11-13T18:26:28.003 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.29954252799999997

Ignoring malparsed backtrace: 2021-11-13T18:26:28.028 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "commit_sent",
, 2021-11-13T18:26:28.028 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:01.585545+0000",
, 2021-11-13T18:26:28.028 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.000310582

Ignoring malparsed backtrace: 2021-11-13T18:26:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "done",
, 2021-11-13T18:26:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:01.681643+0000",
, 2021-11-13T18:26:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00015194000000000001

Ignoring malparsed backtrace: 2021-11-13T18:26:28.076 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.076 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:28.076 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:05.220360+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.104 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.104 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:28.104 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:05.220615+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.132 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.132 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:28.133 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:05.938519+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.161 INFO:teuthology.orchestra.run.smithi183.stdout:                ]
, 2021-11-13T18:26:28.161 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:28.161 INFO:teuthology.orchestra.run.smithi183.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:28.184 INFO:teuthology.orchestra.run.smithi183.stdout:                "events": [
, 2021-11-13T18:26:28.184 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.185 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.207 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.207 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.207 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.229 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.229 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.229 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "op_commit",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops
, 2021-11-13T18:26:28.525+0000 7fed237fe700 -1 mon.a@0(leader).osd e222 definitely_dead 0
, 2021-11-13T18:26:28.584 INFO:teuthology.orchestra.run.smithi183.stdout:{

Ignoring malparsed backtrace: 2021-11-13T18:26:28.609 INFO:teuthology.orchestra.run.smithi183.stdout:                    "client_addr": "172.21.15.157:0/1116373644",
, 2021-11-13T18:26:28.609 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 570
, 2021-11-13T18:26:28.609 INFO:teuthology.orchestra.run.smithi183.stdout:                },

Ignoring malparsed backtrace: 2021-11-13T18:26:28.631 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:28.632 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:54.293776+0000",
, 2021-11-13T18:26:28.632 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 4294967295.9999657

Ignoring malparsed backtrace: 2021-11-13T18:26:28.654 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:28.654 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:14.295832+0000",
, 2021-11-13T18:26:28.654 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.100135276

Ignoring malparsed backtrace: 2021-11-13T18:26:28.676 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "done",
, 2021-11-13T18:26:28.676 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:30.320645+0000",
, 2021-11-13T18:26:28.677 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.0011254349999999999

Ignoring malparsed backtrace: 2021-11-13T18:26:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:45.349390+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.721 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.722 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:28.722 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:45.357284+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.744 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.744 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for subops from 6,7",
, 2021-11-13T18:26:28.744 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:45.462850+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.766 INFO:teuthology.orchestra.run.smithi183.stdout:                ]
, 2021-11-13T18:26:28.766 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:28.767 INFO:teuthology.orchestra.run.smithi183.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:28.788 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.789 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.789 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.811 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.811 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.811 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.832 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.833 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.833 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.855 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.855 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.855 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.883 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.883 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.883 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.905 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.905 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.905 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.928 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.928 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.928 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.950 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.950 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.951 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "op_commit",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.972 INFO:teuthology.orchestra.run.smithi183.stdout:}
, 2021-11-13T18:26:28.973 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub
, 2021-11-13T18:26:28.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight

Ignoring malparsed backtrace: 2021-11-13T18:26:29.308 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:29.308 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:29.309 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-11-13T18:26:29.333 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:29.334 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:29.334 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-11-13T18:26:29.358 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 1.7464000000000001e-05
, 2021-11-13T18:26:29.358 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:29.358 INFO:teuthology.orchestra.run.smithi183.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:29.381 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 22
, 2021-11-13T18:26:29.381 INFO:teuthology.orchestra.run.smithi183.stdout:                },
, 2021-11-13T18:26:29.382 INFO:teuthology.orchestra.run.smithi183.stdout:                "events": [

Ignoring malparsed backtrace: 2021-11-13T18:26:29.404 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:44.211504+0000",
, 2021-11-13T18:26:29.404 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0
, 2021-11-13T18:26:29.404 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.426 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:44.211670+0000",
, 2021-11-13T18:26:29.426 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0
, 2021-11-13T18:26:29.426 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.447 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:47.898234+0000",
, 2021-11-13T18:26:29.447 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.100184361
, 2021-11-13T18:26:29.448 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.469 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.240234+0000",
, 2021-11-13T18:26:29.469 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 7.3578000000000006e-05
, 2021-11-13T18:26:29.469 INFO:teuthology.orchestra.run.smithi183.stdout:                    }

Ignoring malparsed backtrace: 2021-11-13T18:26:29.490 INFO:teuthology.orchestra.run.smithi183.stdout:                    "client_addr": "172.21.15.157:0/1116373644",
, 2021-11-13T18:26:29.491 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 261
, 2021-11-13T18:26:29.491 INFO:teuthology.orchestra.run.smithi183.stdout:                },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.512 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:29.512 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.108951+0000",
, 2021-11-13T18:26:29.512 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 4294967295.9999614

Ignoring malparsed backtrace: 2021-11-13T18:26:29.533 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:29.534 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.109333+0000",
, 2021-11-13T18:26:29.534 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 6.9129999999999997e-06

Ignoring malparsed backtrace: 2021-11-13T18:26:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for subops from 0,7",
, 2021-11-13T18:26:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.654645+0000",
, 2021-11-13T18:26:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.43848754299999998

Ignoring malparsed backtrace: 2021-11-13T18:26:29.577 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:29.577 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:53.761384+0000",
, 2021-11-13T18:26:29.577 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.103891515

Ignoring malparsed backtrace: 2021-11-13T18:26:29.598 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:29.599 INFO:teuthology.orchestra.run.smithi183.stdout:        },
, 2021-11-13T18:26:29.599 INFO:teuthology.orchestra.run.smithi183.stdout:        {

Ignoring malparsed backtrace: 2021-11-13T18:26:29.620 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:29.620 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "throttled",
, 2021-11-13T18:26:29.621 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:04.354142+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:29.990 INFO:teuthology.orchestra.run.smithi183.stdout:            "duration": 1.242249835,
, 2021-11-13T18:26:29.990 INFO:teuthology.orchestra.run.smithi183.stdout:            "type_data": {
, 2021-11-13T18:26:29.991 INFO:teuthology.orchestra.run.smithi183.stdout:                "flag_point": "commit sent; apply or cleanup",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.017 INFO:teuthology.orchestra.run.smithi183.stdout:                    "client_addr": "172.21.15.157:0/2321799831",
, 2021-11-13T18:26:30.018 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 112
, 2021-11-13T18:26:30.018 INFO:teuthology.orchestra.run.smithi183.stdout:                },

Ignoring malparsed backtrace: 2021-11-13T18:26:30.042 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:30.042 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:32.583286+0000",
, 2021-11-13T18:26:30.042 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 4294967295.9999666

Ignoring malparsed backtrace: 2021-11-13T18:26:30.063 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:30.063 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:32.583609+0000",
, 2021-11-13T18:26:30.064 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 9.5559999999999996e-06

Ignoring malparsed backtrace: 2021-11-13T18:26:30.085 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for subops from 0,6",
, 2021-11-13T18:26:30.085 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:32.685029+0000",
, 2021-11-13T18:26:30.086 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00041830900000000001

Ignoring malparsed backtrace: 2021-11-13T18:26:30.107 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:30.107 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:33.392498+0000",
, 2021-11-13T18:26:30.107 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00045502500000000001

Ignoring malparsed backtrace: 2021-11-13T18:26:30.128 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:30.129 INFO:teuthology.orchestra.run.smithi183.stdout:        },
, 2021-11-13T18:26:30.129 INFO:teuthology.orchestra.run.smithi183.stdout:        {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.150 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.150 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "initiated",
, 2021-11-13T18:26:30.150 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:13.283081+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.176 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.176 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "throttled",
, 2021-11-13T18:26:30.176 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:13.283265+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.197 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.197 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:30.198 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.248116+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.558639+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "started",
, 2021-11-13T18:26:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.663596+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.262 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.263 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:30.263 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.967047+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.284 INFO:teuthology.orchestra.run.smithi183.stdout:                ]
, 2021-11-13T18:26:30.284 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:30.285 INFO:teuthology.orchestra.run.smithi183.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:30.306 INFO:teuthology.orchestra.run.smithi183.stdout:                "events": [
, 2021-11-13T18:26:30.306 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.306 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.327 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:30.327 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.327 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.348 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:30.348 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.349 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.369 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:30.370 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.370 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.781 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00044648499999999999
, 2021-11-13T18:26:30.781 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.781 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.816 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 1.098047175
, 2021-11-13T18:26:30.816 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.816 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.839 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.000577507
, 2021-11-13T18:26:30.839 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.840 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.862 INFO:teuthology.orchestra.run.smithi157.stdout:                    }
, 2021-11-13T18:26:30.862 INFO:teuthology.orchestra.run.smithi157.stdout:                ]
, 2021-11-13T18:26:30.862 INFO:teuthology.orchestra.run.smithi157.stdout:            }

Ignoring malparsed backtrace: 2021-11-13T18:26:30.886 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.886 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.886 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.909 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.910 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.910 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.933 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.933 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.933 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.955 INFO:teuthology.orchestra.run.smithi157.stdout:                "client_info": {
, 2021-11-13T18:26:30.955 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client": "client.4439",
, 2021-11-13T18:26:30.955 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client_addr": "172.21.15.157:0/1116373644",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.977 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.977 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.977 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.999 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.999 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.000 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.021 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.021 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.022 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.043 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.044 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.044 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.065 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.066 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.066 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.087 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0
, 2021-11-13T18:26:31.088 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.088 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.110 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 1.5747999999999999e-05
, 2021-11-13T18:26:31.110 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.110 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.132 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 3.941e-06
, 2021-11-13T18:26:31.132 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.132 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.154 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00016938899999999999
, 2021-11-13T18:26:31.155 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.155 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.177 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00012024899999999999
, 2021-11-13T18:26:31.177 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.177 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.200 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00022287399999999999
, 2021-11-13T18:26:31.200 INFO:teuthology.orchestra.run.smithi157.stdout:                    }
, 2021-11-13T18:26:31.200 INFO:teuthology.orchestra.run.smithi157.stdout:                ]

Ignoring malparsed backtrace: 2021-11-13T18:26:45.256 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet
, 2021-11-13T18:26:45.257 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up
, 2021-11-13T18:26:45.257 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

Ignoring malparsed backtrace: 2021-11-13T18:26:55.056 INFO:teuthology.orchestra.run.smithi157.stdout:{
, 2021-11-13T18:26:55.057 INFO:teuthology.orchestra.run.smithi157.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "
, 2021-11-13T18:26:55.057 INFO:teuthology.orchestra.run.smithi157.stdout:}

Ignoring malparsed backtrace: 2021-11-13T18:27:38.264 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-11-13T18:27:38.561 INFO:teuthology.orchestra.run.smithi157.stdout:
, dumped all

Ignoring malparsed backtrace: 2021-11-13T18:27:41.784 DEBUG:teuthology.orchestra.run.smithi157:> mktemp
, 2021-11-13T18:27:41.831 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.4Pp0TeHcJD
, 2021-11-13T18:27:41.831 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.4Pp0TeHcJD

Ignoring malparsed backtrace: 2021-11-13T18:28:01.969 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers
, 2021-11-13T18:28:01.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
, 2021-11-13T18:28:02.000 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-8969-gfedc95a0 (fedc95a0997bbff4576a21a2ec553b013abc1e5b) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7fae79cdd78e]
,  2: /usr/lib/ceph/libceph-common.so.2(+0x27a9a0) [0x7fae79cdd9a0]

Ignoring malparsed backtrace: 2021-11-13T18:26:26.300 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:26.300 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:54.066014+0000",
, 2021-11-13T18:26:26.300 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.20352450999999999

Ignoring malparsed backtrace: 2021-11-13T18:26:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",
, 2021-11-13T18:26:26.328 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.611296+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.353 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.353 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:26.353 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:24:14.889227+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.376 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.376 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "started",
, 2021-11-13T18:26:26.376 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:24:14.994873+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.398 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.399 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:26.399 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:24:15.200041+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.422 INFO:teuthology.orchestra.run.smithi157.stdout:                ]
, 2021-11-13T18:26:26.422 INFO:teuthology.orchestra.run.smithi157.stdout:            }
, 2021-11-13T18:26:26.422 INFO:teuthology.orchestra.run.smithi157.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:26.445 INFO:teuthology.orchestra.run.smithi157.stdout:            "initiated_at": "2021-11-13T18:24:15.900255+0000",
, 2021-11-13T18:26:26.445 INFO:teuthology.orchestra.run.smithi157.stdout:            "age": 130.37645276999999,
, 2021-11-13T18:26:26.446 INFO:teuthology.orchestra.run.smithi157.stdout:            "duration": 0.50350779300000004,

Ignoring malparsed backtrace: 2021-11-13T18:26:26.468 INFO:teuthology.orchestra.run.smithi157.stdout:                "client_info": {
, 2021-11-13T18:26:26.469 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client": "client.4439",
, 2021-11-13T18:26:26.469 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client_addr": "172.21.15.157:0/1116373644",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.491 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.491 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.491 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.514 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.514 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.514 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.537 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.537 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.537 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.560 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.560 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.560 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting_for_map not empty",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.583 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.583 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.583 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "commit_sent",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.605 INFO:teuthology.orchestra.run.smithi157.stdout:                    }
, 2021-11-13T18:26:26.605 INFO:teuthology.orchestra.run.smithi157.stdout:                ]
, 2021-11-13T18:26:26.606 INFO:teuthology.orchestra.run.smithi157.stdout:            }

Ignoring malparsed backtrace: 2021-11-13T18:26:26.628 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 1.0631999999999999e-05
, 2021-11-13T18:26:26.628 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.629 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:26.651 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.000107763
, 2021-11-13T18:26:26.651 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.651 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:26.816 INFO:teuthology.orchestra.run.smithi157.stdout:{
, 2021-11-13T18:26:26.817 INFO:teuthology.orchestra.run.smithi157.stdout:    "ops": [
, 2021-11-13T18:26:26.817 INFO:teuthology.orchestra.run.smithi157.stdout:        {

Ignoring malparsed backtrace: 2021-11-13T18:26:26.851 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.851 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:26.852 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:26:20.352153+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.878 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.878 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.878 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.105 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "op_commit",
, 2021-11-13T18:26:27.105 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:47.796640+0000",
, 2021-11-13T18:26:27.105 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.0046705610000000002

Ignoring malparsed backtrace: 2021-11-13T18:26:27.133 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "commit_sent",
, 2021-11-13T18:26:27.133 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:48.109866+0000",
, 2021-11-13T18:26:27.133 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 6.8777999999999998e-05

Ignoring malparsed backtrace: 2021-11-13T18:26:27.156 INFO:teuthology.orchestra.run.smithi157.stdout:            "age": 218.36948089699999,
, 2021-11-13T18:26:27.156 INFO:teuthology.orchestra.run.smithi157.stdout:            "duration": 0.83512053799999997,
, 2021-11-13T18:26:27.156 INFO:teuthology.orchestra.run.smithi157.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-11-13T18:26:27.178 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.178 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",
, 2021-11-13T18:26:27.178 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:49.653555+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.201 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.201 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "dispatched",
, 2021-11-13T18:26:27.201 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.067104+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.222 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.223 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:27.223 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.072206+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "dispatched",
, 2021-11-13T18:26:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.072310+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.265 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.265 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",
, 2021-11-13T18:26:27.266 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.072389+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.287 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.288 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:27.288 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.612311+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.376 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.376 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.376 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.397 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.397 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.397 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.418 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.418 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.418 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.441 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.441 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.441 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.463 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 4294967295.9999728
, 2021-11-13T18:26:27.463 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.463 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:27.619 INFO:teuthology.orchestra.run.smithi183.stdout:{
, 2021-11-13T18:26:27.619 INFO:teuthology.orchestra.run.smithi183.stdout:    "ops": [],
, 2021-11-13T18:26:27.620 INFO:teuthology.orchestra.run.smithi183.stdout:    "num_ops": 0

Ignoring malparsed backtrace: 2021-11-13T18:26:27.867 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:10.025302+0000",
, 2021-11-13T18:26:27.867 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00064953299999999999
, 2021-11-13T18:26:27.867 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:27.891 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:26.468508+0000",
, 2021-11-13T18:26:27.891 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.112490749
, 2021-11-13T18:26:27.891 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:27.918 INFO:teuthology.orchestra.run.smithi183.stdout:        },
, 2021-11-13T18:26:27.918 INFO:teuthology.orchestra.run.smithi183.stdout:        {
, 2021-11-13T18:26:27.919 INFO:teuthology.orchestra.run.smithi183.stdout:            "description": "osd_op(client.4439.0:4005 4.f 4:f78ab5fc:::smithi15722858-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.948 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "dispatched",
, 2021-11-13T18:26:27.948 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:32.063624+0000",
, 2021-11-13T18:26:27.949 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 3.4173e-05

Ignoring malparsed backtrace: 2021-11-13T18:26:27.974 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:27.974 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:00.279540+0000",
, 2021-11-13T18:26:27.974 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 2.5679999999999998e-06

Ignoring malparsed backtrace: 2021-11-13T18:26:28.002 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:28.002 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:00.686238+0000",
, 2021-11-13T18:26:28.003 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.29954252799999997

Ignoring malparsed backtrace: 2021-11-13T18:26:28.028 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "commit_sent",
, 2021-11-13T18:26:28.028 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:01.585545+0000",
, 2021-11-13T18:26:28.028 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.000310582

Ignoring malparsed backtrace: 2021-11-13T18:26:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "done",
, 2021-11-13T18:26:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:01.681643+0000",
, 2021-11-13T18:26:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00015194000000000001

Ignoring malparsed backtrace: 2021-11-13T18:26:28.076 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.076 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:28.076 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:05.220360+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.104 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.104 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:28.104 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:05.220615+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.132 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.132 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:28.133 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:05.938519+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.161 INFO:teuthology.orchestra.run.smithi183.stdout:                ]
, 2021-11-13T18:26:28.161 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:28.161 INFO:teuthology.orchestra.run.smithi183.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:28.184 INFO:teuthology.orchestra.run.smithi183.stdout:                "events": [
, 2021-11-13T18:26:28.184 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.185 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.207 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.207 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.207 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.229 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.229 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.229 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "op_commit",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops
, 2021-11-13T18:26:28.525+0000 7fed237fe700 -1 mon.a@0(leader).osd e222 definitely_dead 0
, 2021-11-13T18:26:28.584 INFO:teuthology.orchestra.run.smithi183.stdout:{

Ignoring malparsed backtrace: 2021-11-13T18:26:28.609 INFO:teuthology.orchestra.run.smithi183.stdout:                    "client_addr": "172.21.15.157:0/1116373644",
, 2021-11-13T18:26:28.609 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 570
, 2021-11-13T18:26:28.609 INFO:teuthology.orchestra.run.smithi183.stdout:                },

Ignoring malparsed backtrace: 2021-11-13T18:26:28.631 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:28.632 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:54.293776+0000",
, 2021-11-13T18:26:28.632 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 4294967295.9999657

Ignoring malparsed backtrace: 2021-11-13T18:26:28.654 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:28.654 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:14.295832+0000",
, 2021-11-13T18:26:28.654 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.100135276

Ignoring malparsed backtrace: 2021-11-13T18:26:28.676 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "done",
, 2021-11-13T18:26:28.676 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:30.320645+0000",
, 2021-11-13T18:26:28.677 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.0011254349999999999

Ignoring malparsed backtrace: 2021-11-13T18:26:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:45.349390+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.721 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.722 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:28.722 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:45.357284+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.744 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.744 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for subops from 6,7",
, 2021-11-13T18:26:28.744 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:45.462850+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.766 INFO:teuthology.orchestra.run.smithi183.stdout:                ]
, 2021-11-13T18:26:28.766 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:28.767 INFO:teuthology.orchestra.run.smithi183.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:28.788 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.789 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.789 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.811 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.811 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.811 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.832 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.833 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.833 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.855 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.855 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.855 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.883 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.883 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.883 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.905 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.905 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.905 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.928 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.928 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.928 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.950 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.950 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.951 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "op_commit",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.972 INFO:teuthology.orchestra.run.smithi183.stdout:}
, 2021-11-13T18:26:28.973 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub
, 2021-11-13T18:26:28.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight

Ignoring malparsed backtrace: 2021-11-13T18:26:29.308 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:29.308 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:29.309 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-11-13T18:26:29.333 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:29.334 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:29.334 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-11-13T18:26:29.358 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 1.7464000000000001e-05
, 2021-11-13T18:26:29.358 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:29.358 INFO:teuthology.orchestra.run.smithi183.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:29.381 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 22
, 2021-11-13T18:26:29.381 INFO:teuthology.orchestra.run.smithi183.stdout:                },
, 2021-11-13T18:26:29.382 INFO:teuthology.orchestra.run.smithi183.stdout:                "events": [

Ignoring malparsed backtrace: 2021-11-13T18:26:29.404 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:44.211504+0000",
, 2021-11-13T18:26:29.404 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0
, 2021-11-13T18:26:29.404 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.426 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:44.211670+0000",
, 2021-11-13T18:26:29.426 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0
, 2021-11-13T18:26:29.426 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.447 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:47.898234+0000",
, 2021-11-13T18:26:29.447 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.100184361
, 2021-11-13T18:26:29.448 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.469 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.240234+0000",
, 2021-11-13T18:26:29.469 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 7.3578000000000006e-05
, 2021-11-13T18:26:29.469 INFO:teuthology.orchestra.run.smithi183.stdout:                    }

Ignoring malparsed backtrace: 2021-11-13T18:26:29.490 INFO:teuthology.orchestra.run.smithi183.stdout:                    "client_addr": "172.21.15.157:0/1116373644",
, 2021-11-13T18:26:29.491 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 261
, 2021-11-13T18:26:29.491 INFO:teuthology.orchestra.run.smithi183.stdout:                },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.512 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:29.512 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.108951+0000",
, 2021-11-13T18:26:29.512 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 4294967295.9999614

Ignoring malparsed backtrace: 2021-11-13T18:26:29.533 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:29.534 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.109333+0000",
, 2021-11-13T18:26:29.534 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 6.9129999999999997e-06

Ignoring malparsed backtrace: 2021-11-13T18:26:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for subops from 0,7",
, 2021-11-13T18:26:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.654645+0000",
, 2021-11-13T18:26:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.43848754299999998

Ignoring malparsed backtrace: 2021-11-13T18:26:29.577 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:29.577 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:53.761384+0000",
, 2021-11-13T18:26:29.577 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.103891515

Ignoring malparsed backtrace: 2021-11-13T18:26:29.598 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:29.599 INFO:teuthology.orchestra.run.smithi183.stdout:        },
, 2021-11-13T18:26:29.599 INFO:teuthology.orchestra.run.smithi183.stdout:        {

Ignoring malparsed backtrace: 2021-11-13T18:26:29.620 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:29.620 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "throttled",
, 2021-11-13T18:26:29.621 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:04.354142+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:29.990 INFO:teuthology.orchestra.run.smithi183.stdout:            "duration": 1.242249835,
, 2021-11-13T18:26:29.990 INFO:teuthology.orchestra.run.smithi183.stdout:            "type_data": {
, 2021-11-13T18:26:29.991 INFO:teuthology.orchestra.run.smithi183.stdout:                "flag_point": "commit sent; apply or cleanup",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.017 INFO:teuthology.orchestra.run.smithi183.stdout:                    "client_addr": "172.21.15.157:0/2321799831",
, 2021-11-13T18:26:30.018 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 112
, 2021-11-13T18:26:30.018 INFO:teuthology.orchestra.run.smithi183.stdout:                },

Ignoring malparsed backtrace: 2021-11-13T18:26:30.042 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:30.042 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:32.583286+0000",
, 2021-11-13T18:26:30.042 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 4294967295.9999666

Ignoring malparsed backtrace: 2021-11-13T18:26:30.063 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:30.063 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:32.583609+0000",
, 2021-11-13T18:26:30.064 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 9.5559999999999996e-06

Ignoring malparsed backtrace: 2021-11-13T18:26:30.085 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for subops from 0,6",
, 2021-11-13T18:26:30.085 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:32.685029+0000",
, 2021-11-13T18:26:30.086 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00041830900000000001

Ignoring malparsed backtrace: 2021-11-13T18:26:30.107 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:30.107 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:33.392498+0000",
, 2021-11-13T18:26:30.107 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00045502500000000001

Ignoring malparsed backtrace: 2021-11-13T18:26:30.128 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:30.129 INFO:teuthology.orchestra.run.smithi183.stdout:        },
, 2021-11-13T18:26:30.129 INFO:teuthology.orchestra.run.smithi183.stdout:        {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.150 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.150 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "initiated",
, 2021-11-13T18:26:30.150 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:13.283081+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.176 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.176 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "throttled",
, 2021-11-13T18:26:30.176 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:13.283265+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.197 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.197 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:30.198 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.248116+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.558639+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "started",
, 2021-11-13T18:26:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.663596+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.262 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.263 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:30.263 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.967047+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.284 INFO:teuthology.orchestra.run.smithi183.stdout:                ]
, 2021-11-13T18:26:30.284 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:30.285 INFO:teuthology.orchestra.run.smithi183.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:30.306 INFO:teuthology.orchestra.run.smithi183.stdout:                "events": [
, 2021-11-13T18:26:30.306 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.306 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.327 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:30.327 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.327 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.348 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:30.348 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.349 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.369 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:30.370 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.370 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.781 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00044648499999999999
, 2021-11-13T18:26:30.781 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.781 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.816 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 1.098047175
, 2021-11-13T18:26:30.816 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.816 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.839 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.000577507
, 2021-11-13T18:26:30.839 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.840 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.862 INFO:teuthology.orchestra.run.smithi157.stdout:                    }
, 2021-11-13T18:26:30.862 INFO:teuthology.orchestra.run.smithi157.stdout:                ]
, 2021-11-13T18:26:30.862 INFO:teuthology.orchestra.run.smithi157.stdout:            }

Ignoring malparsed backtrace: 2021-11-13T18:26:30.886 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.886 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.886 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.909 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.910 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.910 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.933 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.933 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.933 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.955 INFO:teuthology.orchestra.run.smithi157.stdout:                "client_info": {
, 2021-11-13T18:26:30.955 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client": "client.4439",
, 2021-11-13T18:26:30.955 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client_addr": "172.21.15.157:0/1116373644",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.977 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.977 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.977 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.999 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.999 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.000 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.021 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.021 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.022 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.043 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.044 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.044 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.065 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.066 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.066 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.087 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0
, 2021-11-13T18:26:31.088 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.088 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.110 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 1.5747999999999999e-05
, 2021-11-13T18:26:31.110 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.110 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.132 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 3.941e-06
, 2021-11-13T18:26:31.132 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.132 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.154 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00016938899999999999
, 2021-11-13T18:26:31.155 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.155 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.177 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00012024899999999999
, 2021-11-13T18:26:31.177 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.177 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.200 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00022287399999999999
, 2021-11-13T18:26:31.200 INFO:teuthology.orchestra.run.smithi157.stdout:                    }
, 2021-11-13T18:26:31.200 INFO:teuthology.orchestra.run.smithi157.stdout:                ]

Ignoring malparsed backtrace: 2021-11-13T18:26:45.256 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet
, 2021-11-13T18:26:45.257 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up
, 2021-11-13T18:26:45.257 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

Ignoring malparsed backtrace: 2021-11-13T18:26:55.056 INFO:teuthology.orchestra.run.smithi157.stdout:{
, 2021-11-13T18:26:55.057 INFO:teuthology.orchestra.run.smithi157.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "
, 2021-11-13T18:26:55.057 INFO:teuthology.orchestra.run.smithi157.stdout:}

Ignoring malparsed backtrace: 2021-11-13T18:27:38.264 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-11-13T18:27:38.561 INFO:teuthology.orchestra.run.smithi157.stdout:
, dumped all

Ignoring malparsed backtrace: 2021-11-13T18:27:41.784 DEBUG:teuthology.orchestra.run.smithi157:> mktemp
, 2021-11-13T18:27:41.831 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.4Pp0TeHcJD
, 2021-11-13T18:27:41.831 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.4Pp0TeHcJD

Ignoring malparsed backtrace: 2021-11-13T18:28:01.969 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers
, 2021-11-13T18:28:01.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
, 2021-11-13T18:28:02.000 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-8969-gfedc95a0 (fedc95a0997bbff4576a21a2ec553b013abc1e5b) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7fae79cdd78e]
,  2: /usr/lib/ceph/libceph-common.so.2(+0x27a9a0) [0x7fae79cdd9a0]

Ignoring malparsed backtrace: 2021-11-13T18:26:26.300 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:26.300 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:54.066014+0000",
, 2021-11-13T18:26:26.300 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.20352450999999999

Ignoring malparsed backtrace: 2021-11-13T18:26:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",
, 2021-11-13T18:26:26.328 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.611296+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.353 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.353 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:26.353 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:24:14.889227+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.376 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.376 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "started",
, 2021-11-13T18:26:26.376 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:24:14.994873+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.398 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.399 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:26.399 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:24:15.200041+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.422 INFO:teuthology.orchestra.run.smithi157.stdout:                ]
, 2021-11-13T18:26:26.422 INFO:teuthology.orchestra.run.smithi157.stdout:            }
, 2021-11-13T18:26:26.422 INFO:teuthology.orchestra.run.smithi157.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:26.445 INFO:teuthology.orchestra.run.smithi157.stdout:            "initiated_at": "2021-11-13T18:24:15.900255+0000",
, 2021-11-13T18:26:26.445 INFO:teuthology.orchestra.run.smithi157.stdout:            "age": 130.37645276999999,
, 2021-11-13T18:26:26.446 INFO:teuthology.orchestra.run.smithi157.stdout:            "duration": 0.50350779300000004,

Ignoring malparsed backtrace: 2021-11-13T18:26:26.468 INFO:teuthology.orchestra.run.smithi157.stdout:                "client_info": {
, 2021-11-13T18:26:26.469 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client": "client.4439",
, 2021-11-13T18:26:26.469 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client_addr": "172.21.15.157:0/1116373644",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.491 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.491 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.491 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.514 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.514 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.514 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.537 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.537 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.537 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.560 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.560 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.560 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting_for_map not empty",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.583 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.583 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.583 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "commit_sent",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.605 INFO:teuthology.orchestra.run.smithi157.stdout:                    }
, 2021-11-13T18:26:26.605 INFO:teuthology.orchestra.run.smithi157.stdout:                ]
, 2021-11-13T18:26:26.606 INFO:teuthology.orchestra.run.smithi157.stdout:            }

Ignoring malparsed backtrace: 2021-11-13T18:26:26.628 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 1.0631999999999999e-05
, 2021-11-13T18:26:26.628 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.629 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:26.651 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.000107763
, 2021-11-13T18:26:26.651 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.651 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:26.816 INFO:teuthology.orchestra.run.smithi157.stdout:{
, 2021-11-13T18:26:26.817 INFO:teuthology.orchestra.run.smithi157.stdout:    "ops": [
, 2021-11-13T18:26:26.817 INFO:teuthology.orchestra.run.smithi157.stdout:        {

Ignoring malparsed backtrace: 2021-11-13T18:26:26.851 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.851 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:26.852 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:26:20.352153+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:26.878 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:26.878 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:26.878 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.105 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "op_commit",
, 2021-11-13T18:26:27.105 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:47.796640+0000",
, 2021-11-13T18:26:27.105 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.0046705610000000002

Ignoring malparsed backtrace: 2021-11-13T18:26:27.133 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "commit_sent",
, 2021-11-13T18:26:27.133 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:48.109866+0000",
, 2021-11-13T18:26:27.133 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 6.8777999999999998e-05

Ignoring malparsed backtrace: 2021-11-13T18:26:27.156 INFO:teuthology.orchestra.run.smithi157.stdout:            "age": 218.36948089699999,
, 2021-11-13T18:26:27.156 INFO:teuthology.orchestra.run.smithi157.stdout:            "duration": 0.83512053799999997,
, 2021-11-13T18:26:27.156 INFO:teuthology.orchestra.run.smithi157.stdout:            "type_data": {

Ignoring malparsed backtrace: 2021-11-13T18:26:27.178 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.178 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",
, 2021-11-13T18:26:27.178 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:22:49.653555+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.201 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.201 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "dispatched",
, 2021-11-13T18:26:27.201 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.067104+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.222 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.223 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:27.223 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.072206+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "dispatched",
, 2021-11-13T18:26:27.244 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.072310+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.265 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.265 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",
, 2021-11-13T18:26:27.266 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.072389+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.287 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.288 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:27.288 INFO:teuthology.orchestra.run.smithi157.stdout:                        "time": "2021-11-13T18:23:54.612311+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "header_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.376 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.376 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.376 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.397 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.397 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.397 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.418 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.418 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.418 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.441 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.441 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:27.441 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.463 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 4294967295.9999728
, 2021-11-13T18:26:27.463 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:27.463 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:27.619 INFO:teuthology.orchestra.run.smithi183.stdout:{
, 2021-11-13T18:26:27.619 INFO:teuthology.orchestra.run.smithi183.stdout:    "ops": [],
, 2021-11-13T18:26:27.620 INFO:teuthology.orchestra.run.smithi183.stdout:    "num_ops": 0

Ignoring malparsed backtrace: 2021-11-13T18:26:27.867 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:10.025302+0000",
, 2021-11-13T18:26:27.867 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00064953299999999999
, 2021-11-13T18:26:27.867 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:27.891 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:26.468508+0000",
, 2021-11-13T18:26:27.891 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.112490749
, 2021-11-13T18:26:27.891 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:27.918 INFO:teuthology.orchestra.run.smithi183.stdout:        },
, 2021-11-13T18:26:27.918 INFO:teuthology.orchestra.run.smithi183.stdout:        {
, 2021-11-13T18:26:27.919 INFO:teuthology.orchestra.run.smithi183.stdout:            "description": "osd_op(client.4439.0:4005 4.f 4:f78ab5fc:::smithi15722858-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)",

Ignoring malparsed backtrace: 2021-11-13T18:26:27.948 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "dispatched",
, 2021-11-13T18:26:27.948 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:32.063624+0000",
, 2021-11-13T18:26:27.949 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 3.4173e-05

Ignoring malparsed backtrace: 2021-11-13T18:26:27.974 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:27.974 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:00.279540+0000",
, 2021-11-13T18:26:27.974 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 2.5679999999999998e-06

Ignoring malparsed backtrace: 2021-11-13T18:26:28.002 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:28.002 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:00.686238+0000",
, 2021-11-13T18:26:28.003 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.29954252799999997

Ignoring malparsed backtrace: 2021-11-13T18:26:28.028 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "commit_sent",
, 2021-11-13T18:26:28.028 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:01.585545+0000",
, 2021-11-13T18:26:28.028 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.000310582

Ignoring malparsed backtrace: 2021-11-13T18:26:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "done",
, 2021-11-13T18:26:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:01.681643+0000",
, 2021-11-13T18:26:28.052 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00015194000000000001

Ignoring malparsed backtrace: 2021-11-13T18:26:28.076 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.076 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:28.076 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:05.220360+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.104 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.104 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:28.104 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:05.220615+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.132 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.132 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:28.133 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:26:05.938519+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.161 INFO:teuthology.orchestra.run.smithi183.stdout:                ]
, 2021-11-13T18:26:28.161 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:28.161 INFO:teuthology.orchestra.run.smithi183.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:28.184 INFO:teuthology.orchestra.run.smithi183.stdout:                "events": [
, 2021-11-13T18:26:28.184 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.185 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.207 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.207 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.207 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.229 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.229 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.229 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "op_commit",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops
, 2021-11-13T18:26:28.525+0000 7fed237fe700 -1 mon.a@0(leader).osd e222 definitely_dead 0
, 2021-11-13T18:26:28.584 INFO:teuthology.orchestra.run.smithi183.stdout:{

Ignoring malparsed backtrace: 2021-11-13T18:26:28.609 INFO:teuthology.orchestra.run.smithi183.stdout:                    "client_addr": "172.21.15.157:0/1116373644",
, 2021-11-13T18:26:28.609 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 570
, 2021-11-13T18:26:28.609 INFO:teuthology.orchestra.run.smithi183.stdout:                },

Ignoring malparsed backtrace: 2021-11-13T18:26:28.631 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:28.632 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:54.293776+0000",
, 2021-11-13T18:26:28.632 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 4294967295.9999657

Ignoring malparsed backtrace: 2021-11-13T18:26:28.654 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:28.654 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:14.295832+0000",
, 2021-11-13T18:26:28.654 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.100135276

Ignoring malparsed backtrace: 2021-11-13T18:26:28.676 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "done",
, 2021-11-13T18:26:28.676 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:30.320645+0000",
, 2021-11-13T18:26:28.677 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.0011254349999999999

Ignoring malparsed backtrace: 2021-11-13T18:26:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:28.699 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:45.349390+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.721 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.722 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:28.722 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:45.357284+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.744 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.744 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for subops from 6,7",
, 2021-11-13T18:26:28.744 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:45.462850+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.766 INFO:teuthology.orchestra.run.smithi183.stdout:                ]
, 2021-11-13T18:26:28.766 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:28.767 INFO:teuthology.orchestra.run.smithi183.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:28.788 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.789 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.789 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "dispatched",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.811 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.811 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.811 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.832 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.833 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.833 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.855 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.855 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.855 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.883 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.883 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.883 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.905 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.905 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.905 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.928 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.928 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.928 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for peered",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.950 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:28.950 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:28.951 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "op_commit",

Ignoring malparsed backtrace: 2021-11-13T18:26:28.972 INFO:teuthology.orchestra.run.smithi183.stdout:}
, 2021-11-13T18:26:28.973 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub
, 2021-11-13T18:26:28.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight

Ignoring malparsed backtrace: 2021-11-13T18:26:29.308 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:29.308 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:29.309 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "throttled",

Ignoring malparsed backtrace: 2021-11-13T18:26:29.333 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:29.334 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:29.334 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "done",

Ignoring malparsed backtrace: 2021-11-13T18:26:29.358 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 1.7464000000000001e-05
, 2021-11-13T18:26:29.358 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:29.358 INFO:teuthology.orchestra.run.smithi183.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:29.381 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 22
, 2021-11-13T18:26:29.381 INFO:teuthology.orchestra.run.smithi183.stdout:                },
, 2021-11-13T18:26:29.382 INFO:teuthology.orchestra.run.smithi183.stdout:                "events": [

Ignoring malparsed backtrace: 2021-11-13T18:26:29.404 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:44.211504+0000",
, 2021-11-13T18:26:29.404 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0
, 2021-11-13T18:26:29.404 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.426 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:44.211670+0000",
, 2021-11-13T18:26:29.426 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0
, 2021-11-13T18:26:29.426 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.447 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:47.898234+0000",
, 2021-11-13T18:26:29.447 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.100184361
, 2021-11-13T18:26:29.448 INFO:teuthology.orchestra.run.smithi183.stdout:                    },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.469 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.240234+0000",
, 2021-11-13T18:26:29.469 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 7.3578000000000006e-05
, 2021-11-13T18:26:29.469 INFO:teuthology.orchestra.run.smithi183.stdout:                    }

Ignoring malparsed backtrace: 2021-11-13T18:26:29.490 INFO:teuthology.orchestra.run.smithi183.stdout:                    "client_addr": "172.21.15.157:0/1116373644",
, 2021-11-13T18:26:29.491 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 261
, 2021-11-13T18:26:29.491 INFO:teuthology.orchestra.run.smithi183.stdout:                },

Ignoring malparsed backtrace: 2021-11-13T18:26:29.512 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:29.512 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.108951+0000",
, 2021-11-13T18:26:29.512 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 4294967295.9999614

Ignoring malparsed backtrace: 2021-11-13T18:26:29.533 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:29.534 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.109333+0000",
, 2021-11-13T18:26:29.534 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 6.9129999999999997e-06

Ignoring malparsed backtrace: 2021-11-13T18:26:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for subops from 0,7",
, 2021-11-13T18:26:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:49.654645+0000",
, 2021-11-13T18:26:29.555 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.43848754299999998

Ignoring malparsed backtrace: 2021-11-13T18:26:29.577 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:29.577 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:53.761384+0000",
, 2021-11-13T18:26:29.577 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.103891515

Ignoring malparsed backtrace: 2021-11-13T18:26:29.598 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:29.599 INFO:teuthology.orchestra.run.smithi183.stdout:        },
, 2021-11-13T18:26:29.599 INFO:teuthology.orchestra.run.smithi183.stdout:        {

Ignoring malparsed backtrace: 2021-11-13T18:26:29.620 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:29.620 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "throttled",
, 2021-11-13T18:26:29.621 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:25:04.354142+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:29.990 INFO:teuthology.orchestra.run.smithi183.stdout:            "duration": 1.242249835,
, 2021-11-13T18:26:29.990 INFO:teuthology.orchestra.run.smithi183.stdout:            "type_data": {
, 2021-11-13T18:26:29.991 INFO:teuthology.orchestra.run.smithi183.stdout:                "flag_point": "commit sent; apply or cleanup",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.017 INFO:teuthology.orchestra.run.smithi183.stdout:                    "client_addr": "172.21.15.157:0/2321799831",
, 2021-11-13T18:26:30.018 INFO:teuthology.orchestra.run.smithi183.stdout:                    "tid": 112
, 2021-11-13T18:26:30.018 INFO:teuthology.orchestra.run.smithi183.stdout:                },

Ignoring malparsed backtrace: 2021-11-13T18:26:30.042 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "header_read",
, 2021-11-13T18:26:30.042 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:32.583286+0000",
, 2021-11-13T18:26:30.042 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 4294967295.9999666

Ignoring malparsed backtrace: 2021-11-13T18:26:30.063 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:30.063 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:32.583609+0000",
, 2021-11-13T18:26:30.064 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 9.5559999999999996e-06

Ignoring malparsed backtrace: 2021-11-13T18:26:30.085 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "waiting for subops from 0,6",
, 2021-11-13T18:26:30.085 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:32.685029+0000",
, 2021-11-13T18:26:30.086 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00041830900000000001

Ignoring malparsed backtrace: 2021-11-13T18:26:30.107 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",
, 2021-11-13T18:26:30.107 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:22:33.392498+0000",
, 2021-11-13T18:26:30.107 INFO:teuthology.orchestra.run.smithi183.stdout:                        "duration": 0.00045502500000000001

Ignoring malparsed backtrace: 2021-11-13T18:26:30.128 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:30.129 INFO:teuthology.orchestra.run.smithi183.stdout:        },
, 2021-11-13T18:26:30.129 INFO:teuthology.orchestra.run.smithi183.stdout:        {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.150 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.150 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "initiated",
, 2021-11-13T18:26:30.150 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:13.283081+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.176 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.176 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "throttled",
, 2021-11-13T18:26:30.176 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:13.283265+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.197 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.197 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "queued_for_pg",
, 2021-11-13T18:26:30.198 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.248116+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:30.219 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.558639+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "started",
, 2021-11-13T18:26:30.241 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.663596+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.262 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.263 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",
, 2021-11-13T18:26:30.263 INFO:teuthology.orchestra.run.smithi183.stdout:                        "time": "2021-11-13T18:23:24.967047+0000",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.284 INFO:teuthology.orchestra.run.smithi183.stdout:                ]
, 2021-11-13T18:26:30.284 INFO:teuthology.orchestra.run.smithi183.stdout:            }
, 2021-11-13T18:26:30.285 INFO:teuthology.orchestra.run.smithi183.stdout:        },

Ignoring malparsed backtrace: 2021-11-13T18:26:30.306 INFO:teuthology.orchestra.run.smithi183.stdout:                "events": [
, 2021-11-13T18:26:30.306 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.306 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "initiated",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.327 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:30.327 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.327 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.348 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:30.348 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.349 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.369 INFO:teuthology.orchestra.run.smithi183.stdout:                    },
, 2021-11-13T18:26:30.370 INFO:teuthology.orchestra.run.smithi183.stdout:                    {
, 2021-11-13T18:26:30.370 INFO:teuthology.orchestra.run.smithi183.stdout:                        "event": "sub_op_commit_rec",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.781 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00044648499999999999
, 2021-11-13T18:26:30.781 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.781 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.816 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 1.098047175
, 2021-11-13T18:26:30.816 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.816 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.839 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.000577507
, 2021-11-13T18:26:30.839 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.840 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:30.862 INFO:teuthology.orchestra.run.smithi157.stdout:                    }
, 2021-11-13T18:26:30.862 INFO:teuthology.orchestra.run.smithi157.stdout:                ]
, 2021-11-13T18:26:30.862 INFO:teuthology.orchestra.run.smithi157.stdout:            }

Ignoring malparsed backtrace: 2021-11-13T18:26:30.886 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.886 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.886 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.909 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.910 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.910 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.933 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.933 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.933 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.955 INFO:teuthology.orchestra.run.smithi157.stdout:                "client_info": {
, 2021-11-13T18:26:30.955 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client": "client.4439",
, 2021-11-13T18:26:30.955 INFO:teuthology.orchestra.run.smithi157.stdout:                    "client_addr": "172.21.15.157:0/1116373644",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.977 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.977 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:30.977 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "all_read",

Ignoring malparsed backtrace: 2021-11-13T18:26:30.999 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:30.999 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.000 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "waiting for rw locks",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.021 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.021 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.022 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.043 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.044 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.044 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "started",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.065 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.066 INFO:teuthology.orchestra.run.smithi157.stdout:                    {
, 2021-11-13T18:26:31.066 INFO:teuthology.orchestra.run.smithi157.stdout:                        "event": "reached_pg",

Ignoring malparsed backtrace: 2021-11-13T18:26:31.087 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0
, 2021-11-13T18:26:31.088 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.088 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.110 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 1.5747999999999999e-05
, 2021-11-13T18:26:31.110 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.110 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.132 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 3.941e-06
, 2021-11-13T18:26:31.132 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.132 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.154 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00016938899999999999
, 2021-11-13T18:26:31.155 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.155 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.177 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00012024899999999999
, 2021-11-13T18:26:31.177 INFO:teuthology.orchestra.run.smithi157.stdout:                    },
, 2021-11-13T18:26:31.177 INFO:teuthology.orchestra.run.smithi157.stdout:                    {

Ignoring malparsed backtrace: 2021-11-13T18:26:31.200 INFO:teuthology.orchestra.run.smithi157.stdout:                        "duration": 0.00022287399999999999
, 2021-11-13T18:26:31.200 INFO:teuthology.orchestra.run.smithi157.stdout:                    }
, 2021-11-13T18:26:31.200 INFO:teuthology.orchestra.run.smithi157.stdout:                ]

Ignoring malparsed backtrace: 2021-11-13T18:26:45.256 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet
, 2021-11-13T18:26:45.257 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up
, 2021-11-13T18:26:45.257 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

Ignoring malparsed backtrace: 2021-11-13T18:26:55.056 INFO:teuthology.orchestra.run.smithi157.stdout:{
, 2021-11-13T18:26:55.057 INFO:teuthology.orchestra.run.smithi157.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "
, 2021-11-13T18:26:55.057 INFO:teuthology.orchestra.run.smithi157.stdout:}

Ignoring malparsed backtrace: 2021-11-13T18:27:38.264 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2021-11-13T18:27:38.561 INFO:teuthology.orchestra.run.smithi157.stdout:
, dumped all

Ignoring malparsed backtrace: 2021-11-13T18:27:41.784 DEBUG:teuthology.orchestra.run.smithi157:> mktemp
, 2021-11-13T18:27:41.831 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.4Pp0TeHcJD
, 2021-11-13T18:27:41.831 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.4Pp0TeHcJD

Ignoring malparsed backtrace: 2021-11-13T18:28:01.969 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers
, 2021-11-13T18:28:01.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
, 2021-11-13T18:28:02.000 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-8969-gfedc95a0 (fedc95a0997bbff4576a21a2ec553b013abc1e5b) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x561b1c0eafe2]
,  2: ceph-osd(+0x5d4203) [0x561b1c0eb203]

Ignoring malparsed backtrace:  ceph version 17.0.0-8969-gfedc95a0 (fedc95a0997bbff4576a21a2ec553b013abc1e5b) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x561b1c0eafe2]
,  2: ceph-osd(+0x5d4203) [0x561b1c0eb203]

Ignoring malparsed backtrace:  ceph version 17.0.0-8969-gfedc95a0 (fedc95a0997bbff4576a21a2ec553b013abc1e5b) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x561b1c0eafe2]
,  2: ceph-osd(+0x5d4203) [0x561b1c0eb203]

Found 2 distinct failure reasons
Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0'
1 jobs: ['6501848']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/dedup-io-snaps}']

Timeout 120 running ceph --cluster ceph osd dump --format=json
1 jobs: ['6501834']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/set-chunks-read}']