2021-02-28T07:16:59.176 INFO:root:teuthology version: 1.1.0-3082387b 2021-02-28T07:16:59.179 INFO:teuthology.results:subset: '0/999' 2021-02-28T07:16:59.180 INFO:teuthology.results:seed: '9951' 2021-02-28T07:37:09.105 INFO:root:teuthology version: 1.1.0-3082387b 2021-02-28T07:37:09.141 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-02-28T08:10:11.563 INFO:teuthology.results:Tests finished! gathering results... 2021-02-28T08:10:12.070 INFO:scrape:Found 8 jobs 2021-02-28T08:10:12.334 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1308-g7f59c9b8 (7f59c9b8a0bbe7c55db2f518ba5f553acb089c36) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fadff17d33d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fadff17d518] 2021-02-28T08:10:12.335 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4080000000000001e-05 , 2021-02-28T07:41:18.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2021-02-28T08:10:12.335 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.488798+0000", , 2021-02-28T07:41:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 , 2021-02-28T07:41:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.697355+0000", , 2021-02-28T07:41:18.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.20847777200000001 , 2021-02-28T07:41:18.741 INFO:teuthology.orchestra.run.smithi060.stdout: } 2021-02-28T08:10:12.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", , 2021-02-28T07:41:18.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.604175+0000", , 2021-02-28T07:41:18.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049184200000000004 2021-02-28T08:10:12.337 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.789 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:18.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", , 2021-02-28T07:41:18.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.636228+0000", 2021-02-28T08:10:12.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.812 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2021-02-28T07:40:26.636470+0000", , 2021-02-28T07:41:18.812 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 52.04264611, , 2021-02-28T07:41:18.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.061410613000000003, 2021-02-28T08:10:12.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.837 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:18.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2021-02-28T08:10:12.339 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8779999999999997e-06 , 2021-02-28T07:41:18.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2021-02-28T08:10:12.340 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.883 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5509 , 2021-02-28T07:41:18.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.884 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2021-02-28T08:10:12.340 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:32.306523+0000", , 2021-02-28T07:41:18.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 , 2021-02-28T07:41:18.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", , 2021-02-28T07:41:18.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:39.687142+0000", , 2021-02-28T07:41:18.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2021-02-28T08:10:12.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.957 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 27.552437136000002, , 2021-02-28T07:41:18.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.035251847000000003, , 2021-02-28T07:41:18.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2021-02-28T08:10:12.342 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:25.373990+0000", , 2021-02-28T07:41:19.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026075859999999998 , 2021-02-28T07:41:19.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.343 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.402 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.403 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(osd.3.0:438 4.0 e31/20 4:03c71143:::smithi02521556-62:head v 31'120, mlcod=29'118)", 2021-02-28T08:10:12.343 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", , 2021-02-28T07:41:19.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:32.288606+0000", , 2021-02-28T07:41:19.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9382999999999998e-05 2021-02-28T08:10:12.344 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.451 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", , 2021-02-28T07:41:19.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:36.532693+0000", 2021-02-28T08:10:12.344 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: ] , 2021-02-28T07:41:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: } , 2021-02-28T07:41:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.345 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.500 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2021-02-28T08:10:12.346 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7458e-05 , 2021-02-28T07:41:19.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2021-02-28T08:10:12.346 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:42.125701+0000", , 2021-02-28T07:41:19.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999886 , 2021-02-28T07:41:19.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.347 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0053848380000000003, , 2021-02-28T07:41:19.571 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { , 2021-02-28T07:41:19.571 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-28T08:10:12.347 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.594 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", , 2021-02-28T07:41:19.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:46.405494+0000", , 2021-02-28T07:41:19.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042795619999999998 2021-02-28T08:10:12.348 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.618 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", , 2021-02-28T07:41:19.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:50.882974+0000", 2021-02-28T08:10:12.349 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.642 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2021-02-28T08:10:12.351 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:36.103 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 , 2021-02-28T07:41:36.221+0000 7fd55542f700 -1 osd.0 63 handle_osd_map injecting map crc failure , reweighted osd.6 to 1 (10000) 2021-02-28T08:10:12.352 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:49.495 INFO:teuthology.orchestra.run.smithi025.stdout:} , 2021-02-28T07:41:49.509 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 , instructed osd(s) 1 to deep-scrub 2021-02-28T08:10:12.353 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:42:34.281 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/contextutil.py", line 33, in nested 2021-02-28T08:10:12.353 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:42:37.255 DEBUG:teuthology.orchestra.run.smithi025:> rm -fr /tmp/tmp.3bBWo4GBJi , 2021-02-28T07:42:37.261 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/myoungwon-2021-02-28_07:11:46-rados-wip-manifest-dedup-test-distro-basic-smithi/5920660/data/mon.b.tgz , 2021-02-28T07:42:37.262 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2021-02-28T08:10:12.354 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:42:51.808 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core , 2021-02-28T07:42:51.821 DEBUG:teuthology.orchestra.run.smithi025:> test -e /home/ubuntu/cephtest/archive/coredump , 2021-02-28T07:42:51.845 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi025.front.sepia.ceph.com, flagging run as failed 2021-02-28T08:10:12.432 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1308-g7f59c9b8 (7f59c9b8a0bbe7c55db2f518ba5f553acb089c36) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fadff17d33d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fadff17d518] 2021-02-28T08:10:12.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4080000000000001e-05 , 2021-02-28T07:41:18.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2021-02-28T08:10:12.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.488798+0000", , 2021-02-28T07:41:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 , 2021-02-28T07:41:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.697355+0000", , 2021-02-28T07:41:18.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.20847777200000001 , 2021-02-28T07:41:18.741 INFO:teuthology.orchestra.run.smithi060.stdout: } 2021-02-28T08:10:12.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", , 2021-02-28T07:41:18.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.604175+0000", , 2021-02-28T07:41:18.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049184200000000004 2021-02-28T08:10:12.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.789 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:18.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", , 2021-02-28T07:41:18.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.636228+0000", 2021-02-28T08:10:12.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.812 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2021-02-28T07:40:26.636470+0000", , 2021-02-28T07:41:18.812 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 52.04264611, , 2021-02-28T07:41:18.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.061410613000000003, 2021-02-28T08:10:12.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.837 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:18.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2021-02-28T08:10:12.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8779999999999997e-06 , 2021-02-28T07:41:18.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2021-02-28T08:10:12.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.883 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5509 , 2021-02-28T07:41:18.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.884 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2021-02-28T08:10:12.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:32.306523+0000", , 2021-02-28T07:41:18.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 , 2021-02-28T07:41:18.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", , 2021-02-28T07:41:18.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:39.687142+0000", , 2021-02-28T07:41:18.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2021-02-28T08:10:12.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.957 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 27.552437136000002, , 2021-02-28T07:41:18.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.035251847000000003, , 2021-02-28T07:41:18.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2021-02-28T08:10:12.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:25.373990+0000", , 2021-02-28T07:41:19.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026075859999999998 , 2021-02-28T07:41:19.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.402 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.403 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(osd.3.0:438 4.0 e31/20 4:03c71143:::smithi02521556-62:head v 31'120, mlcod=29'118)", 2021-02-28T08:10:12.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", , 2021-02-28T07:41:19.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:32.288606+0000", , 2021-02-28T07:41:19.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9382999999999998e-05 2021-02-28T08:10:12.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.451 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", , 2021-02-28T07:41:19.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:36.532693+0000", 2021-02-28T08:10:12.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: ] , 2021-02-28T07:41:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: } , 2021-02-28T07:41:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.500 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2021-02-28T08:10:12.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7458e-05 , 2021-02-28T07:41:19.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2021-02-28T08:10:12.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:42.125701+0000", , 2021-02-28T07:41:19.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999886 , 2021-02-28T07:41:19.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.446 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0053848380000000003, , 2021-02-28T07:41:19.571 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { , 2021-02-28T07:41:19.571 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-28T08:10:12.446 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.594 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", , 2021-02-28T07:41:19.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:46.405494+0000", , 2021-02-28T07:41:19.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042795619999999998 2021-02-28T08:10:12.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.618 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", , 2021-02-28T07:41:19.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:50.882974+0000", 2021-02-28T08:10:12.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.642 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2021-02-28T08:10:12.449 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:36.103 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 , 2021-02-28T07:41:36.221+0000 7fd55542f700 -1 osd.0 63 handle_osd_map injecting map crc failure , reweighted osd.6 to 1 (10000) 2021-02-28T08:10:12.450 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:49.495 INFO:teuthology.orchestra.run.smithi025.stdout:} , 2021-02-28T07:41:49.509 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 , instructed osd(s) 1 to deep-scrub 2021-02-28T08:10:12.451 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:42:34.281 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/contextutil.py", line 33, in nested 2021-02-28T08:10:12.452 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:42:37.255 DEBUG:teuthology.orchestra.run.smithi025:> rm -fr /tmp/tmp.3bBWo4GBJi , 2021-02-28T07:42:37.261 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/myoungwon-2021-02-28_07:11:46-rados-wip-manifest-dedup-test-distro-basic-smithi/5920660/data/mon.b.tgz , 2021-02-28T07:42:37.262 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2021-02-28T08:10:12.452 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:42:51.808 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core , 2021-02-28T07:42:51.821 DEBUG:teuthology.orchestra.run.smithi025:> test -e /home/ubuntu/cephtest/archive/coredump , 2021-02-28T07:42:51.845 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi025.front.sepia.ceph.com, flagging run as failed 2021-02-28T08:10:12.534 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1308-g7f59c9b8 (7f59c9b8a0bbe7c55db2f518ba5f553acb089c36) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fadff17d33d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fadff17d518] 2021-02-28T08:10:12.535 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4080000000000001e-05 , 2021-02-28T07:41:18.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2021-02-28T08:10:12.536 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.488798+0000", , 2021-02-28T07:41:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 , 2021-02-28T07:41:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.536 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.697355+0000", , 2021-02-28T07:41:18.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.20847777200000001 , 2021-02-28T07:41:18.741 INFO:teuthology.orchestra.run.smithi060.stdout: } 2021-02-28T08:10:12.537 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", , 2021-02-28T07:41:18.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.604175+0000", , 2021-02-28T07:41:18.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00049184200000000004 2021-02-28T08:10:12.537 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.789 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:18.789 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", , 2021-02-28T07:41:18.789 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:26.636228+0000", 2021-02-28T08:10:12.538 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.812 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2021-02-28T07:40:26.636470+0000", , 2021-02-28T07:41:18.812 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 52.04264611, , 2021-02-28T07:41:18.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.061410613000000003, 2021-02-28T08:10:12.539 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.837 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:18.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2021-02-28T08:10:12.540 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8779999999999997e-06 , 2021-02-28T07:41:18.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2021-02-28T08:10:12.540 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.883 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 5509 , 2021-02-28T07:41:18.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:18.884 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2021-02-28T08:10:12.541 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:32.306523+0000", , 2021-02-28T07:41:18.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9609999999999999e-06 , 2021-02-28T07:41:18.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.541 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", , 2021-02-28T07:41:18.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:39.687142+0000", , 2021-02-28T07:41:18.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4905e-05 2021-02-28T08:10:12.542 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:18.957 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 27.552437136000002, , 2021-02-28T07:41:18.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.035251847000000003, , 2021-02-28T07:41:18.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2021-02-28T08:10:12.543 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:25.373990+0000", , 2021-02-28T07:41:19.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026075859999999998 , 2021-02-28T07:41:19.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.543 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.402 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.403 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(osd.3.0:438 4.0 e31/20 4:03c71143:::smithi02521556-62:head v 31'120, mlcod=29'118)", 2021-02-28T08:10:12.544 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", , 2021-02-28T07:41:19.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:32.288606+0000", , 2021-02-28T07:41:19.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9382999999999998e-05 2021-02-28T08:10:12.545 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.451 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", , 2021-02-28T07:41:19.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:36.532693+0000", 2021-02-28T08:10:12.545 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: ] , 2021-02-28T07:41:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: } , 2021-02-28T07:41:19.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.546 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.500 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2021-02-28T08:10:12.547 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7458e-05 , 2021-02-28T07:41:19.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2021-02-28T08:10:12.547 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:42.125701+0000", , 2021-02-28T07:41:19.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4294967295.9999886 , 2021-02-28T07:41:19.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2021-02-28T08:10:12.548 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0053848380000000003, , 2021-02-28T07:41:19.571 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { , 2021-02-28T07:41:19.571 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-28T08:10:12.549 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.594 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", , 2021-02-28T07:41:19.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:46.405494+0000", , 2021-02-28T07:41:19.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042795619999999998 2021-02-28T08:10:12.549 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.618 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", , 2021-02-28T07:41:19.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2021-02-28T07:40:50.882974+0000", 2021-02-28T08:10:12.550 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, , 2021-02-28T07:41:19.642 INFO:teuthology.orchestra.run.smithi060.stdout: { , 2021-02-28T07:41:19.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2021-02-28T08:10:12.552 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:36.103 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 , 2021-02-28T07:41:36.221+0000 7fd55542f700 -1 osd.0 63 handle_osd_map injecting map crc failure , reweighted osd.6 to 1 (10000) 2021-02-28T08:10:12.553 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:41:49.495 INFO:teuthology.orchestra.run.smithi025.stdout:} , 2021-02-28T07:41:49.509 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 , instructed osd(s) 1 to deep-scrub 2021-02-28T08:10:12.554 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:42:34.281 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/contextutil.py", line 33, in nested 2021-02-28T08:10:12.555 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:42:37.255 DEBUG:teuthology.orchestra.run.smithi025:> rm -fr /tmp/tmp.3bBWo4GBJi , 2021-02-28T07:42:37.261 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/myoungwon-2021-02-28_07:11:46-rados-wip-manifest-dedup-test-distro-basic-smithi/5920660/data/mon.b.tgz , 2021-02-28T07:42:37.262 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2021-02-28T08:10:12.555 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:42:51.808 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core , 2021-02-28T07:42:51.821 DEBUG:teuthology.orchestra.run.smithi025:> test -e /home/ubuntu/cephtest/archive/coredump , 2021-02-28T07:42:51.845 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi025.front.sepia.ceph.com, flagging run as failed 2021-02-28T08:10:12.692 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1308-g7f59c9b8 (7f59c9b8a0bbe7c55db2f518ba5f553acb089c36) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f6ca255933d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f6ca2559518] 2021-02-28T08:10:12.693 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:45:54.614 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2021-02-28T07:45:54.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops , 2021-02-28T07:45:54.626 DEBUG:teuthology.orchestra.run.smithi137:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op apply-layout-settings --pool unique_pool_0 2021-02-28T08:10:12.696 WARNING:scrape:Ignoring malparsed backtrace: set osd.6 primary-affinity to 1 (8655362) , 2021-02-28T07:46:24.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 , reweighted osd.2 to 1 (10000) 2021-02-28T08:10:12.698 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:46:38.727 INFO:tasks.ceph:Scrubbing osd.2 , 2021-02-28T07:46:38.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2021-02-28T07:46:38.890 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-02-28T08:10:12.700 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:46:42.049 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2021-02-28T07:46:29.347668+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=28, tm_hour=7, tm_min=46, tm_sec=29, tm_wday=6, tm_yday=59, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=28, tm_hour=7, tm_min=46, tm_sec=36, tm_wday=6, tm_yday=59, tm_isdst=0) , 2021-02-28T07:46:42.049 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2021-02-28T07:46:45.027+0000 7fe3bcb92700 -1 osd.5 pg_epoch: 115 pg[1.1( empty local-lis/les=92/93 n=0 ec=60/15 lis/c=92/92 les/c/f=93/93/0 sis=92) [5,0,2] r=0 lpr=92 crt=0'0 mlcod 0'0 active+clean MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber pg(1.1) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2 (0x55dc15ca83c0) 2021-02-28T08:10:12.701 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:47:50.057 INFO:teuthology.misc:Shutting down mgr daemons... , 2021-02-28T07:47:50.058 DEBUG:tasks.ceph.mgr.y:waiting for process to exit , 2021-02-28T07:47:50.059 INFO:teuthology.orchestra.run:waiting for 300 2021-02-28T08:10:12.701 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 179, in _raise_for_status 2021-02-28T08:10:12.788 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1308-g7f59c9b8 (7f59c9b8a0bbe7c55db2f518ba5f553acb089c36) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fa3f5e3d33d] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fa3f5e3d518] 2021-02-28T08:10:12.789 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:26.859 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", , 2021-02-28T07:52:26.859 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:43.972589+0000", , 2021-02-28T07:52:26.859 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6682999999999998e-05 2021-02-28T08:10:12.789 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:26.880 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 42.783011107, , 2021-02-28T07:52:26.880 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.052099022000000002, , 2021-02-28T07:52:26.881 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2021-02-28T08:10:12.790 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:26.901 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:26.902 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", , 2021-02-28T07:52:26.902 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:44.077848+0000", 2021-02-28T08:10:12.791 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:26.922 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:26.922 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:26.923 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2021-02-28T08:10:12.791 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:26.943 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.4479.0:2393 3.0 e25/18 3:0bea4576:::smithi12921495-146 low_tier:head v 25'32, mlcod=25'28)", , 2021-02-28T07:52:26.943 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2021-02-28T07:51:44.309486+0000", , 2021-02-28T07:52:26.943 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 42.527527323000001, 2021-02-28T08:10:12.792 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:26.964 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5132000000000002e-05 , 2021-02-28T07:52:26.964 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:26.964 INFO:teuthology.orchestra.run.smithi129.stdout: { 2021-02-28T08:10:12.792 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:26.985 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:44.658334+0000", , 2021-02-28T07:52:26.985 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003078084 , 2021-02-28T07:52:26.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2021-02-28T08:10:12.793 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.006 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:27.006 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.007 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.4479.0:2536 3.0 e25/18 3:0d9f4f07:::smithi12921495-289 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 25'64, mlcod=25'61)", 2021-02-28T08:10:12.794 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.028 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", , 2021-02-28T07:52:27.028 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:45.339887+0000", , 2021-02-28T07:52:27.028 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8483000000000003e-05 2021-02-28T08:10:12.794 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.050 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.050 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", , 2021-02-28T07:52:27.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:45.554945+0000", 2021-02-28T08:10:12.795 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.070 INFO:teuthology.orchestra.run.smithi129.stdout: ] , 2021-02-28T07:52:27.070 INFO:teuthology.orchestra.run.smithi129.stdout: } , 2021-02-28T07:52:27.071 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2021-02-28T08:10:12.796 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.091 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:27.091 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.091 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2021-02-28T08:10:12.796 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.580 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", , 2021-02-28T07:52:27.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:43.857702+0000", 2021-02-28T08:10:12.797 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.602 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", , 2021-02-28T07:52:27.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:43.857781+0000", 2021-02-28T08:10:12.797 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.623 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.623 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", , 2021-02-28T07:52:27.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:43.862776+0000", 2021-02-28T08:10:12.798 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.644 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", , 2021-02-28T07:52:27.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:44.211874+0000", 2021-02-28T08:10:12.799 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.666 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.666 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", , 2021-02-28T07:52:27.666 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:51:44.263618+0000", 2021-02-28T08:10:12.799 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.687 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { , 2021-02-28T07:52:27.688 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.4479", , 2021-02-28T07:52:27.688 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "v1:172.21.15.129:0/4106464982", 2021-02-28T08:10:12.800 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.709 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:27.709 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.709 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2021-02-28T08:10:12.800 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.731 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:27.731 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.731 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2021-02-28T08:10:12.801 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.753 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:27.753 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.753 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 5,6", 2021-02-28T08:10:12.802 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.775 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:27.775 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.775 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2021-02-28T08:10:12.802 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.796 INFO:teuthology.orchestra.run.smithi129.stdout: } , 2021-02-28T07:52:27.796 INFO:teuthology.orchestra.run.smithi129.stdout: ] , 2021-02-28T07:52:27.796 INFO:teuthology.orchestra.run.smithi129.stdout: } 2021-02-28T08:10:12.803 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.818 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:27.818 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ , 2021-02-28T07:52:27.818 INFO:teuthology.orchestra.run.smithi129.stdout: { 2021-02-28T08:10:12.803 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.839 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4085000000000001e-05 , 2021-02-28T07:52:27.840 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:27.840 INFO:teuthology.orchestra.run.smithi129.stdout: { 2021-02-28T08:10:12.804 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.861 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20460500100000001 , 2021-02-28T07:52:27.861 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:52:27.861 INFO:teuthology.orchestra.run.smithi129.stdout: { 2021-02-28T08:10:12.805 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.883 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:52:27.883 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.4479.0:7704 3.2 3:5b9d1804:::smithi12921495-62 low_tier:head [write 1628354~709677] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e48)", , 2021-02-28T07:52:27.883 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2021-02-28T07:52:23.232857+0000", 2021-02-28T08:10:12.805 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.904 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:52:23.233439+0000", , 2021-02-28T07:52:27.905 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2050000000000001e-06 , 2021-02-28T07:52:27.905 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2021-02-28T08:10:12.806 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:27.926 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2021-02-28T07:52:23.236887+0000", , 2021-02-28T07:52:27.926 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6806999999999998e-05 , 2021-02-28T07:52:27.927 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2021-02-28T08:10:12.806 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:28.334 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:28.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2021-02-28T08:10:12.807 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:28.356 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:28.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2021-02-28T08:10:12.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.377 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4479.0:2534 3.3 3:f19cff77:::smithi12921495-287 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 64] snapc 0=[] ondisk+write+known_if_redirected e25)", , 2021-02-28T07:52:28.378 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2021-02-28T07:51:44.946898+0000", , 2021-02-28T07:52:28.378 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 43.369085505000001, 2021-02-28T08:10:12.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 , 2021-02-28T07:52:28.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:28.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.809 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5617e-05 , 2021-02-28T07:52:28.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:28.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.810 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029035200000000001 , 2021-02-28T07:52:28.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:28.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.810 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.043903769000000002 , 2021-02-28T07:52:28.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:28.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.811 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3343999999999997e-05 , 2021-02-28T07:52:28.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:28.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.811 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { , 2021-02-28T07:52:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-02-28T07:52:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2021-02-28T08:10:12.812 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:54.860392+0000", , 2021-02-28T07:52:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001902525 , 2021-02-28T07:52:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2021-02-28T08:10:12.813 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:54.865411+0000", , 2021-02-28T07:52:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000173776 , 2021-02-28T07:52:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2021-02-28T08:10:12.813 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:54.970680+0000", , 2021-02-28T07:52:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.108278164 , 2021-02-28T07:52:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: } 2021-02-28T08:10:12.814 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", , 2021-02-28T07:52:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:52:17.659246+0000", , 2021-02-28T07:52:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3700000000000003e-06 2021-02-28T08:10:12.814 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", , 2021-02-28T07:52:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:52:23.231575+0000", 2021-02-28T08:10:12.815 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", , 2021-02-28T07:52:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:52:23.436007+0000", 2021-02-28T08:10:12.816 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:29.052 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:29.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2021-02-28T08:10:12.816 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:29.079 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:29.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2021-02-28T08:10:12.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2869999999999993e-06 , 2021-02-28T07:52:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:44.175500+0000", , 2021-02-28T07:52:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 , 2021-02-28T07:52:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2021-02-28T08:10:12.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:44.240999+0000", , 2021-02-28T07:52:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5077999999999999e-05 , 2021-02-28T07:52:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2021-02-28T08:10:12.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", , 2021-02-28T07:52:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:44.215142+0000", , 2021-02-28T07:52:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8882999999999999e-05 2021-02-28T08:10:12.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", , 2021-02-28T07:52:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:44.226361+0000", 2021-02-28T08:10:12.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", , 2021-02-28T07:52:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:44.601479+0000", 2021-02-28T08:10:12.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:29.244 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:29.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2021-02-28T08:10:12.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.267 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-02-28T07:52:29.267 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ , 2021-02-28T07:52:29.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.822 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.064972641999999997 , 2021-02-28T07:52:29.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:29.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.822 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:45.317595+0000", , 2021-02-28T07:52:29.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8971000000000001e-05 , 2021-02-28T07:52:29.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2021-02-28T08:10:12.823 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.471 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], , 2021-02-28T07:52:29.471 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 , 2021-02-28T07:52:29.471 INFO:teuthology.orchestra.run.smithi192.stdout:} 2021-02-28T08:10:12.824 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:43.788416+0000", , 2021-02-28T07:52:29.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4255e-05 , 2021-02-28T07:52:29.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2021-02-28T08:10:12.824 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.075223469000000001, , 2021-02-28T07:52:29.773 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { , 2021-02-28T07:52:29.773 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-28T08:10:12.825 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", , 2021-02-28T07:52:29.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:44.550351+0000", , 2021-02-28T07:52:29.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.082003276999999999 2021-02-28T08:10:12.826 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.813 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:29.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", , 2021-02-28T07:52:29.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:44.559335+0000", 2021-02-28T08:10:12.826 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.835 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2021-02-28T07:51:45.041443+0000", , 2021-02-28T07:52:29.835 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 44.689006663999997, , 2021-02-28T07:52:29.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.080069013999999994, 2021-02-28T08:10:12.827 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:29.856 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:29.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2021-02-28T08:10:12.827 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015580030000000001 , 2021-02-28T07:52:29.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:29.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.828 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.897 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:29.897 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4479.0:2559 3.3 e25/18 3:c12d8ef6:::smithi12921495-312 low_tier:head v 25'66, mlcod=25'61)", , 2021-02-28T07:52:29.898 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2021-02-28T07:51:45.077260+0000", 2021-02-28T08:10:12.829 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:45.085036+0000", , 2021-02-28T07:52:29.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4573e-05 , 2021-02-28T07:52:29.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2021-02-28T08:10:12.829 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", , 2021-02-28T07:52:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:45.087221+0000", , 2021-02-28T07:52:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023256380000000001 2021-02-28T08:10:12.830 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: } , 2021-02-28T07:52:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.831 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", , 2021-02-28T07:52:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:45.316338+0000", 2021-02-28T08:10:12.831 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:30.396 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:30.396 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4479.0:1062 4.8 e23/19 4:1ead8aec:::smithi12921495-236:head v 23'44, mlcod=23'42)", 2021-02-28T08:10:12.832 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", , 2021-02-28T07:52:30.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:49.297989+0000", , 2021-02-28T07:52:30.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0326999999999999e-05 2021-02-28T08:10:12.832 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.440 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:30.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", , 2021-02-28T07:52:30.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:51:53.316313+0000", 2021-02-28T08:10:12.833 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.461 INFO:teuthology.orchestra.run.smithi192.stdout: ] , 2021-02-28T07:52:30.461 INFO:teuthology.orchestra.run.smithi192.stdout: } , 2021-02-28T07:52:30.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2021-02-28T08:10:12.834 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:30.483 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:30.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2021-02-28T08:10:12.834 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5754999999999999e-05 , 2021-02-28T07:52:30.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:30.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2021-02-28T08:10:12.835 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.526 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { , 2021-02-28T07:52:30.526 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-02-28T07:52:30.526 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2021-02-28T08:10:12.835 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:52:14.192464+0000", , 2021-02-28T07:52:30.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011104543999999999 , 2021-02-28T07:52:30.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2021-02-28T08:10:12.836 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", , 2021-02-28T07:52:30.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:52:18.218274+0000", , 2021-02-28T07:52:30.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3354e-05 2021-02-28T08:10:12.837 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.593 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:30.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", , 2021-02-28T07:52:30.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2021-02-28T07:52:23.795991+0000", 2021-02-28T08:10:12.837 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.614 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2021-02-28T07:52:23.816272+0000", , 2021-02-28T07:52:30.615 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.5623290189999999, , 2021-02-28T07:52:30.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018123174999999998, 2021-02-28T08:10:12.838 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, , 2021-02-28T07:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: { , 2021-02-28T07:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2021-02-28T08:10:12.839 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:52:51.391 INFO:teuthology.orchestra.run.smithi192.stdout:Write #4:e6fdecd2:::smithi12921495-237:head# , 2021-02-28T07:52:51.391 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} , 2021-02-28T07:52:51.391 INFO:teuthology.orchestra.run.smithi192.stdout:Write #4:e71290a8:::smithi12921495-7:head# 2021-02-28T08:10:12.840 WARNING:scrape:Ignoring malparsed backtrace: no valid command found; 10 closest matches: , 0 , 1 2021-02-28T08:10:12.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:53:51.705 INFO:teuthology.orchestra.run.smithi129.stdout: "erasure_code_profile": "", , 2021-02-28T07:53:51.705 INFO:teuthology.orchestra.run.smithi129.stdout: "hit_set_params": { , 2021-02-28T07:53:51.705 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "none" 2021-02-28T08:10:12.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:53:51.727 INFO:teuthology.orchestra.run.smithi129.stdout: "type": 1, , 2021-02-28T07:53:51.727 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 3, , 2021-02-28T07:53:51.727 INFO:teuthology.orchestra.run.smithi129.stdout: "min_size": 2, 2021-02-28T08:10:12.842 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:53:51.749 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_epoch": 0, , 2021-02-28T07:53:51.749 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_snaps": [], , 2021-02-28T07:53:51.749 INFO:teuthology.orchestra.run.smithi129.stdout: "removed_snaps": "[]", 2021-02-28T08:10:12.842 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:53:51.770 INFO:teuthology.orchestra.run.smithi129.stdout: "in": 1, , 2021-02-28T07:53:51.771 INFO:teuthology.orchestra.run.smithi129.stdout: "weight": 1, , 2021-02-28T07:53:51.771 INFO:teuthology.orchestra.run.smithi129.stdout: "primary_affinity": 1, 2021-02-28T08:10:12.843 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:53:51.793 INFO:teuthology.orchestra.run.smithi129.stdout: ] , 2021-02-28T07:53:51.793 INFO:teuthology.orchestra.run.smithi129.stdout: }, , 2021-02-28T07:53:51.793 INFO:teuthology.orchestra.run.smithi129.stdout: "public_addr": "172.21.15.129:6808/16928", 2021-02-28T08:10:12.844 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:53:51.814 INFO:teuthology.orchestra.run.smithi129.stdout: "addrvec": [ , 2021-02-28T07:53:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:53:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v1", 2021-02-28T08:10:12.844 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:53:51.836 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.192:6800", , 2021-02-28T07:53:51.836 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 16993 , 2021-02-28T07:53:51.836 INFO:teuthology.orchestra.run.smithi129.stdout: } 2021-02-28T08:10:12.845 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:53:51.857 INFO:teuthology.orchestra.run.smithi129.stdout: { , 2021-02-28T07:53:51.857 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": 0, , 2021-02-28T07:53:51.858 INFO:teuthology.orchestra.run.smithi129.stdout: "down_stamp": "0.000000", 2021-02-28T08:10:12.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:53:51.879 INFO:teuthology.orchestra.run.smithi129.stdout: "stretch_mode": { , 2021-02-28T07:53:51.879 INFO:teuthology.orchestra.run.smithi129.stdout: "stretch_mode_enabled": false, , 2021-02-28T07:53:51.879 INFO:teuthology.orchestra.run.smithi129.stdout: "stretch_bucket_count": 0, 2021-02-28T08:10:12.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:54:19.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.2 , 2021-02-28T07:54:19.767 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2021-02-28T07:54:20.068 INFO:teuthology.orchestra.run.smithi129.stdout:55834574887 2021-02-28T08:10:12.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:54:28.593 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2021-02-28T07:54:05.864120+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=28, tm_hour=7, tm_min=54, tm_sec=5, tm_wday=6, tm_yday=59, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=28, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=6, tm_yday=59, tm_isdst=0) , 2021-02-28T07:54:28.593 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2021-02-28T07:54:09.767870+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=28, tm_hour=7, tm_min=54, tm_sec=9, tm_wday=6, tm_yday=59, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=28, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=6, tm_yday=59, tm_isdst=0) , 2021-02-28T07:54:28.594 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2021-02-28T07:53:58.353968+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=28, tm_hour=7, tm_min=53, tm_sec=58, tm_wday=6, tm_yday=59, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=28, tm_hour=7, tm_min=54, tm_sec=23, tm_wday=6, tm_yday=59, tm_isdst=0) 2021-02-28T08:10:12.850 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:55:10.282 INFO:teuthology.orchestra.run:waiting for 300 , 2021-02-28T07:55:10.315 INFO:tasks.ceph.mon.a:Stopped , 2021-02-28T07:55:10.316 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2021-02-28T08:10:12.851 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-28T07:55:32.337 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , 2021-02-28T07:55:32.358 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter , 2021-02-28T07:55:32.358 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2021-02-28T08:10:12.864 INFO:scrape:Found 1 distinct failure reasons 2021-02-28T08:10:12.865 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 50 --op write 50 --op delete 10 --op tier_promote 30 --op write_excl 50 --pool unique_pool_0' 2021-02-28T08:10:12.866 INFO:scrape:3 jobs: ['5920660', '5920663', '5920664'] 2021-02-28T08:10:12.866 INFO:scrape:suites intersection: ['backoff/normal', 'ceph', 'clusters/{fixed-2', 'msgr-failures/fastclose', 'msgr/async-v1only', 'openstack}', 'rados', 'supported-random-distro$/{ubuntu_latest}', 'thrashosds-health', 'workloads/redirect_promote_tests}'] 2021-02-28T08:10:12.867 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/redirect_promote_tests}'] 2021-02-28T08:10:12.867 INFO:scrape: